Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..................X......X...........................] duration: 16.934681136s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 16.934717274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.............X......X.............] duration: 6.103621487s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:49:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.103650464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..................X......X..........................] duration: 16.138008568s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 16.13803592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........................X.............X......X............] duration: 5.838496157s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 5.838523179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..................X......X.........................] duration: 15.317294423s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.317321023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..................X......X........................] duration: 26.130049188s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 26.130084123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........................X.............X......X...........] duration: 6.061742273s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:47:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.061761839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..................X......X.......................] duration: 24.596292973s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 24.596312862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........................X.............X......X..........] duration: 6.204778262s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:46:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.204817365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........................X.............X......X.........] duration: 20.935093541s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 20.935138707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..................X......X......................] duration: 27.978868016s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 27.978895409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........................X.............X......X........] duration: 7.243088643s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 7.243114571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..................X......X.....................] duration: 15.440481705s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.440503466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........................X.............X......X.......] duration: 6.168102902s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.168127518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.........................X.............X......X......] duration: 7.212354238s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 7.21238138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..................X......X....................] duration: 15.349779351s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:44:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.349796244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..................X......X...................] duration: 25.325874954s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:43:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 25.32589361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.........................X.............X......X.....] duration: 6.321415858s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:43:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.321502423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..................X......X..................] duration: 45.517686574s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:43:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 45.517715528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.........................X.............X......X....] duration: 6.327092475s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:42:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.327143853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.........................X.............X......X...] duration: 5.745798244s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:41:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 5.745821821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..................X......X.................] duration: 15.377862925s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.37790201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.........................X.............X......X..] duration: 5.697313513s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:41:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 5.697332358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..................X......X................] duration: 15.387205618s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:41:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.387254872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.........................X.............X......X.] duration: 27.131168735s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:41:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 27.131185479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..................X......X...............] duration: 25.552622155s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:40:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 25.552652121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.........................X.............X......X] duration: 30.003525477s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:40:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 30.00354851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..................X......X..............] duration: 30.794147999s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 30.79417482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.........................X.............X......] duration: 8.160190964s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:39:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 8.160221041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........................X.............X.....] duration: 6.005435983s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:38:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 6.005450079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..................X......X.............] duration: 23.919363553s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:38:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 23.919391006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........................X.............X....] duration: 7.198753996s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:38:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 7.198778884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.........................X.............X...] duration: 11.417300103s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 11.417337965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..................X......X............] duration: 37.30117078s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:37:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 37.301248397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.........................X.............X..] duration: 11.737302399s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 11.737327355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..................X......X...........] duration: 20.707169382s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:36:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 20.707188639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..................X......X..........] duration: 36.426833694s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 36.426871927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.........................X.............X.] duration: 6.204737407s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:35:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 6.204762124s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.........................X.............X] duration: 30.003068231s trace: https://jaeger.dev.storj.io/trace/280968d74ee1e794 FAIL (2025-12-29 22:34:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA) context deadline exceeded ======LOGS====== Dec 29 22:34:16.983396371 (583h5m16.854479406s) [upload=511131]: Starting upload Dec 29 22:34:16.983468238 (583h5m16.854551273s) [upload=511131]: Done starting upload Dec 29 22:34:16.983606752 (583h5m16.854689787s) [upload=511131]: Uploading segments... Dec 29 22:34:16.983723545 (583h5m16.85480658s) [upload=511131]: Got next segment. Inline: false Dec 29 22:34:16.983777668 (583h5m16.854860713s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:34:16.983784891 (583h5m16.854867936s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:34:17.248994532 (583h5m17.120077577s) [seg_pos={0 0}, upload=511131]: Begin upload segment... Dec 29 22:34:17.249060338 (583h5m17.120143373s) [seg_pos={0 0}, upload=511131]: Done begin upload segment. Dec 29 22:34:17.249150119 (583h5m17.120233154s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249171449 (583h5m17.120254484s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249751103 (583h5m17.120834138s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.249757285 (583h5m17.12084032s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250169029 (583h5m17.121252064s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250224745 (583h5m17.12130779s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250565193 (583h5m17.121648228s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.250639775 (583h5m17.12172282s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251046149 (583h5m17.122129184s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251219309 (583h5m17.122302344s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251378061 (583h5m17.122461096s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.251646242 (583h5m17.122729287s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252094455 (583h5m17.12317749s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252545945 (583h5m17.12362899s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.252873228 (583h5m17.123956273s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:17.256509392 (583h5m17.127592427s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Uploading piece... Dec 29 22:34:18.937245301 (583h5m18.808328346s) [upload=511131]: Next returned nil segment Dec 29 22:34:18.937277923 (583h5m18.808360968s) [upload=511131]: Waiting for error group managing segments... Dec 29 22:34:19.219035533 (583h5m19.090118578s) [noise=true , address=145.79.227.6:4259 , piece_id=A6246L3BWFWZKCAUVUU722LJTDJGCBBFPQTIQVVKB6GIUUVP2KZQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.227751469 (583h5m19.098834514s) [noise=true , address=145.79.227.37:4567 , piece_id=MAAZOG5XA47XWNWOF76HJSVELQHQWLJPHHTKQVJTTS3MQKUL6RRA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.228845943 (583h5m19.099928988s) [noise=true , address=145.79.227.45:4231 , piece_id=YIB72UZNV7I2LDMLIW4TKYOXAZW5I5RSL5FD4XSLBHCWK5QKLPQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.240265117 (583h5m19.111348162s) [noise=true , address=145.79.227.48:4303 , piece_id=SRI3GNQ6TVL43JZ756N2IULV5VRK26UMD3MEJUB5R2RSTMAFBAXQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.241057705 (583h5m19.11214074s) [noise=true , address=145.79.227.47:4265 , piece_id=PEZYMFA3YXD6P3DGM6OR2HPXBHBS2VG4WCXIF4GEJWH5OZVBCKDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.252002195 (583h5m19.12308523s) [noise=true , address=145.79.227.46:4602 , piece_id=U6JWUCLMNI7IF5U7Y5AM7YAE4ZLACI6NH5G372HYE4HUCWTWUMOQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.254630770 (583h5m19.125713805s) [noise=true , address=145.79.227.18:4007 , piece_id=TGMPUV5RQIXUPBNF7R435LRNB6IUGLRV2PWLIVTQP7V33XMP3VBQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.416580732 (583h5m19.287663777s) [noise=true , address=77.90.31.17:4770 , piece_id=3TKRCSYZUGBOKCWHRRS3EBOOOFW3CDRM2HZOJJSGWT6FVCTUNHQQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.432679286 (583h5m19.303762321s) [noise=true , address=77.90.31.13:4626 , piece_id=NOIKZVC6LBMRB4HC3F7TW7GBIJ5TITCVHVFZGEEM6VU7WES5CGGQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.496009615 (583h5m19.36709265s) [noise=true , address=45.91.116.7:4011 , piece_id=DSNTIK6LYNPFERXBLHFZENNIYWK4KJYV7WN2LJSXE52PY6RQPNQA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.610751898 (583h5m19.481834943s) [noise=true , address=45.91.116.9:4262 , piece_id=BQSLBMGJISOQZ2NSDSBHCYH2M6QZZH63T6RX53BKGQUK3YOM2G7Q, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:19.619716638 (583h5m19.490799673s) [noise=true , address=45.91.116.5:4297 , piece_id=4XEYSB6FL6MWCTUPN5UDFODZ2F2CI4DNHIJWSLQ67K22Z2I7FLDQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576790398 (583h5m20.447873433s) [noise=true , address=145.79.227.35:4483 , piece_id=6PXCWBQPTDQT7JCM6EJD3W754LWLIWVUQ53JBIYOPW5FG6OMQNVA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: <nil> Dec 29 22:34:20.576808953 (583h5m20.447891988s) [seg_pos={0 0}, upload=511131]: Segment reached optimal threshold of 13 pieces. Dec 29 22:34:20.576953678 (583h5m20.448036723s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.576961353 (583h5m20.448044388s) [noise=true , address=145.79.227.4:4187 , piece_id=BTNVXEQSJ3T5M62XYBTQOV2FHFXTFPAUJ25L44GIILQVPAFUMUYA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577014895 (583h5m20.44809794s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577022950 (583h5m20.448105985s) [noise=true , address=145.79.227.19:4078 , piece_id=55TKS2YJNYDMOU5M44OEF777BGC5OOQ7WGJQBLSZSJHU4ZYWALPQ, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577146285 (583h5m20.44822932s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 22:34:20.577152066 (583h5m20.448235111s) [noise=true , address=77.90.31.12:4589 , piece_id=FDIY6E6NUPCBB62UAGNZ7OQMK7E2OZNE7IGEFRTAKJUNUWWTV7BA, seg_pos={0 0}, upload=511131]: Long tail cancellation due to optimal threshold Dec 29 22:34:20.577168678 (583h5m20.448251723s) [seg_pos={0 0}, upload=511131]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:34:20.577231517 (583h5m20.448314552s) [upload=511131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:34:20.577236397 (583h5m20.448319432s) [upload=511131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:34:20.874035166 (583h5m20.745118221s) [upload=511131]: Done uploading segments... Dec 29 22:34:20.874059041 (583h5m20.745142076s) [upload=511131]: Upload finished. err: <nil> Duration: 30.003092076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..................X......X.........] duration: 15.190668096s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:34:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.190683915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..................X......X........] duration: 16.995798143s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:34:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 16.995837639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........................X.............] duration: 22.219310564s trace: https://jaeger.dev.storj.io/trace/50cd070c446f9532 FAIL (2025-12-29 22:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA) context deadline exceeded ======LOGS====== Dec 29 22:20:38.003755724 (582h51m37.874838759s) [upload=510934]: Starting upload Dec 29 22:20:38.003809547 (582h51m37.874892592s) [upload=510934]: Done starting upload Dec 29 22:20:38.003861455 (582h51m37.87494449s) [upload=510934]: Uploading segments... Dec 29 22:20:38.003995491 (582h51m37.875078526s) [upload=510934]: Got next segment. Inline: false Dec 29 22:20:38.004039846 (582h51m37.875122881s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:20:38.004047089 (582h51m37.875130124s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:20:38.253256798 (582h51m38.124339843s) [seg_pos={0 0}, upload=510934]: Begin upload segment... Dec 29 22:20:38.253324216 (582h51m38.124407251s) [seg_pos={0 0}, upload=510934]: Done begin upload segment. Dec 29 22:20:38.253359373 (582h51m38.124442418s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.253389000 (582h51m38.124472045s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.254399654 (582h51m38.125482689s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255022320 (582h51m38.126105385s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255623524 (582h51m38.126706569s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255639395 (582h51m38.12672243s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256173752 (582h51m38.127256797s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256278241 (582h51m38.127361276s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256604232 (582h51m38.127687267s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256837957 (582h51m38.127920992s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257074728 (582h51m38.128157773s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257379468 (582h51m38.128462513s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257566414 (582h51m38.128649459s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258046438 (582h51m38.129129473s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258587398 (582h51m38.129670433s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.259062062 (582h51m38.130145097s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:39.938508372 (582h51m39.809591407s) [upload=510934]: Next returned nil segment Dec 29 22:20:39.938519794 (582h51m39.809602829s) [upload=510934]: Waiting for error group managing segments... Dec 29 22:20:40.236728734 (582h51m40.107811779s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.238046043 (582h51m40.109129078s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.246341979 (582h51m40.117425024s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.247622127 (582h51m40.118705162s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.251962321 (582h51m40.123045356s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.255370801 (582h51m40.126453836s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256021209 (582h51m40.127104244s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256716132 (582h51m40.127799167s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.275478619 (582h51m40.146561654s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.287513205 (582h51m40.15859624s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.427649182 (582h51m40.298732217s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.510504820 (582h51m40.381587875s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544136709 (582h51m40.415219744s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544156237 (582h51m40.415239282s) [seg_pos={0 0}, upload=510934]: Segment reached optimal threshold of 13 pieces. Dec 29 22:20:40.544344815 (582h51m40.41542785s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544351499 (582h51m40.415434544s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544505341 (582h51m40.415588387s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544546820 (582h51m40.415629865s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544584402 (582h51m40.415667437s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544590794 (582h51m40.415673829s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544641000 (582h51m40.415724035s) [seg_pos={0 0}, upload=510934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:20:40.544714600 (582h51m40.415797635s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:20:40.544719519 (582h51m40.415802554s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:20:40.845710266 (582h51m40.716793301s) [upload=510934]: Done uploading segments... Dec 29 22:20:40.845727149 (582h51m40.716810184s) [upload=510934]: Upload finished. err: <nil> Duration: 22.219336584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..................X......X.......] duration: 15.473597642s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.473619513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..................X......X......] duration: 15.423614493s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.423633138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.........................X............] duration: 5.98742131s trace: https://jaeger.dev.storj.io/trace/50cd070c446f9532 FAIL (2025-12-29 22:32:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA) context deadline exceeded ======LOGS====== Dec 29 22:20:38.003755724 (582h51m37.874838759s) [upload=510934]: Starting upload Dec 29 22:20:38.003809547 (582h51m37.874892592s) [upload=510934]: Done starting upload Dec 29 22:20:38.003861455 (582h51m37.87494449s) [upload=510934]: Uploading segments... Dec 29 22:20:38.003995491 (582h51m37.875078526s) [upload=510934]: Got next segment. Inline: false Dec 29 22:20:38.004039846 (582h51m37.875122881s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:20:38.004047089 (582h51m37.875130124s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:20:38.253256798 (582h51m38.124339843s) [seg_pos={0 0}, upload=510934]: Begin upload segment... Dec 29 22:20:38.253324216 (582h51m38.124407251s) [seg_pos={0 0}, upload=510934]: Done begin upload segment. Dec 29 22:20:38.253359373 (582h51m38.124442418s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.253389000 (582h51m38.124472045s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.254399654 (582h51m38.125482689s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255022320 (582h51m38.126105385s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255623524 (582h51m38.126706569s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255639395 (582h51m38.12672243s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256173752 (582h51m38.127256797s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256278241 (582h51m38.127361276s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256604232 (582h51m38.127687267s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256837957 (582h51m38.127920992s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257074728 (582h51m38.128157773s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257379468 (582h51m38.128462513s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257566414 (582h51m38.128649459s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258046438 (582h51m38.129129473s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258587398 (582h51m38.129670433s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.259062062 (582h51m38.130145097s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:39.938508372 (582h51m39.809591407s) [upload=510934]: Next returned nil segment Dec 29 22:20:39.938519794 (582h51m39.809602829s) [upload=510934]: Waiting for error group managing segments... Dec 29 22:20:40.236728734 (582h51m40.107811779s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.238046043 (582h51m40.109129078s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.246341979 (582h51m40.117425024s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.247622127 (582h51m40.118705162s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.251962321 (582h51m40.123045356s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.255370801 (582h51m40.126453836s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256021209 (582h51m40.127104244s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256716132 (582h51m40.127799167s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.275478619 (582h51m40.146561654s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.287513205 (582h51m40.15859624s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.427649182 (582h51m40.298732217s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.510504820 (582h51m40.381587875s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544136709 (582h51m40.415219744s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544156237 (582h51m40.415239282s) [seg_pos={0 0}, upload=510934]: Segment reached optimal threshold of 13 pieces. Dec 29 22:20:40.544344815 (582h51m40.41542785s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544351499 (582h51m40.415434544s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544505341 (582h51m40.415588387s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544546820 (582h51m40.415629865s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544584402 (582h51m40.415667437s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544590794 (582h51m40.415673829s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544641000 (582h51m40.415724035s) [seg_pos={0 0}, upload=510934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:20:40.544714600 (582h51m40.415797635s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:20:40.544719519 (582h51m40.415802554s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:20:40.845710266 (582h51m40.716793301s) [upload=510934]: Done uploading segments... Dec 29 22:20:40.845727149 (582h51m40.716810184s) [upload=510934]: Upload finished. err: <nil> Duration: 5.987446938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..................X......X.....] duration: 15.208047486s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.208063176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X............................X..................X......X....] duration: 38.008228417s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 38.008250779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.........................X...........] duration: 11.650637148s trace: https://jaeger.dev.storj.io/trace/50cd070c446f9532 FAIL (2025-12-29 22:31:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA) context deadline exceeded ======LOGS====== Dec 29 22:20:38.003755724 (582h51m37.874838759s) [upload=510934]: Starting upload Dec 29 22:20:38.003809547 (582h51m37.874892592s) [upload=510934]: Done starting upload Dec 29 22:20:38.003861455 (582h51m37.87494449s) [upload=510934]: Uploading segments... Dec 29 22:20:38.003995491 (582h51m37.875078526s) [upload=510934]: Got next segment. Inline: false Dec 29 22:20:38.004039846 (582h51m37.875122881s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:20:38.004047089 (582h51m37.875130124s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:20:38.253256798 (582h51m38.124339843s) [seg_pos={0 0}, upload=510934]: Begin upload segment... Dec 29 22:20:38.253324216 (582h51m38.124407251s) [seg_pos={0 0}, upload=510934]: Done begin upload segment. Dec 29 22:20:38.253359373 (582h51m38.124442418s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.253389000 (582h51m38.124472045s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.254399654 (582h51m38.125482689s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255022320 (582h51m38.126105385s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255623524 (582h51m38.126706569s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255639395 (582h51m38.12672243s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256173752 (582h51m38.127256797s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256278241 (582h51m38.127361276s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256604232 (582h51m38.127687267s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256837957 (582h51m38.127920992s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257074728 (582h51m38.128157773s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257379468 (582h51m38.128462513s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257566414 (582h51m38.128649459s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258046438 (582h51m38.129129473s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258587398 (582h51m38.129670433s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.259062062 (582h51m38.130145097s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:39.938508372 (582h51m39.809591407s) [upload=510934]: Next returned nil segment Dec 29 22:20:39.938519794 (582h51m39.809602829s) [upload=510934]: Waiting for error group managing segments... Dec 29 22:20:40.236728734 (582h51m40.107811779s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.238046043 (582h51m40.109129078s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.246341979 (582h51m40.117425024s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.247622127 (582h51m40.118705162s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.251962321 (582h51m40.123045356s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.255370801 (582h51m40.126453836s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256021209 (582h51m40.127104244s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256716132 (582h51m40.127799167s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.275478619 (582h51m40.146561654s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.287513205 (582h51m40.15859624s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.427649182 (582h51m40.298732217s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.510504820 (582h51m40.381587875s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544136709 (582h51m40.415219744s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544156237 (582h51m40.415239282s) [seg_pos={0 0}, upload=510934]: Segment reached optimal threshold of 13 pieces. Dec 29 22:20:40.544344815 (582h51m40.41542785s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544351499 (582h51m40.415434544s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544505341 (582h51m40.415588387s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544546820 (582h51m40.415629865s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544584402 (582h51m40.415667437s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544590794 (582h51m40.415673829s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544641000 (582h51m40.415724035s) [seg_pos={0 0}, upload=510934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:20:40.544714600 (582h51m40.415797635s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:20:40.544719519 (582h51m40.415802554s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:20:40.845710266 (582h51m40.716793301s) [upload=510934]: Done uploading segments... Dec 29 22:20:40.845727149 (582h51m40.716810184s) [upload=510934]: Upload finished. err: <nil> Duration: 11.650663639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X............................X..................X......X...] duration: 15.106315261s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.106334166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X............................X..................X......X..] duration: 27.051545744s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:30:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 27.051570702s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)