Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.............X......X...............X.............] duration: 6.103416525s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 23:01:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.10344042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 15.33477985s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 23:01:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.334809687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.............X......X...............X............] duration: 6.304733371s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 23:00:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.304761494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 21.638474294s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 23:00:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 21.638505194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.............X......X...............X...........] duration: 6.196468079s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 23:00:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.196500241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 47.053083727s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 47.053113213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.............X......X...............X..........] duration: 6.08390371s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:59:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.083923797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............X......X...............X.........] duration: 19.232556528s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:59:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 19.232683931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......X........................................] duration: 15.400692754s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.400708924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration: 35.327222396s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:58:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 35.327238266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.............X......X...............X........] duration: 5.796493276s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:58:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 5.796519407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.............X......X...............X.......] duration: 9.397565707s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:57:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 9.397587549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 15.392247276s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.39228614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.............X......X...............X......] duration: 6.131325379s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:56:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.131350817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.............X......X...............X.....] duration: 6.121507317s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:56:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.121532446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration: 21.546428967s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:56:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 21.546531123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......X....................................] duration: 15.458185515s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:56:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.458216494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.............X......X...............X....] duration: 21.991401615s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:55:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 21.991426953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......X...................................] duration: 15.405036312s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:55:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.405085847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 27.200908927s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:54:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 27.200937422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.............X......X...............X...] duration: 20.92449507s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:54:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 20.924523656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..................X......X.................................] duration: 15.366005072s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.366020552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..................X......X................................] duration: 35.586240059s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 35.586269785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.............X......X...............X..] duration: 7.349525591s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:53:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 7.349553685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.............X......X...............X.] duration: 6.235400389s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:52:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 6.235428564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..................X......X...............................] duration: 17.019678015s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:52:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 17.019711169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..................X......X..............................] duration: 15.841269443s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:51:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.841295804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.............X......X...............X] duration: 30.002608646s trace: https://jaeger.dev.storj.io/trace/14568c7dcb5007bf FAIL (2025-12-29 22:51:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 22:51:23.142113207 (583h22m23.013196262s) [upload=511377]: Starting upload Dec 29 22:51:23.142199111 (583h22m23.013282146s) [upload=511377]: Done starting upload Dec 29 22:51:23.142285836 (583h22m23.013368881s) [upload=511377]: Uploading segments... Dec 29 22:51:23.142438636 (583h22m23.013521682s) [upload=511377]: Got next segment. Inline: false Dec 29 22:51:23.142476660 (583h22m23.013559695s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:51:23.142481188 (583h22m23.013564233s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:51:23.383637500 (583h22m23.254720535s) [seg_pos={0 0}, upload=511377]: Begin upload segment... Dec 29 22:51:23.383693346 (583h22m23.254776381s) [seg_pos={0 0}, upload=511377]: Done begin upload segment. Dec 29 22:51:23.383754252 (583h22m23.254837297s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.383759271 (583h22m23.254842316s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384315080 (583h22m23.255398115s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.384750900 (583h22m23.255833935s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.385131774 (583h22m23.256214819s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386071904 (583h22m23.25715494s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386445446 (583h22m23.257528491s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386686415 (583h22m23.25776945s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.386987909 (583h22m23.258070954s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.387379926 (583h22m23.258462961s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388176933 (583h22m23.259259968s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.388562127 (583h22m23.259645162s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.389749818 (583h22m23.260832863s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.391150996 (583h22m23.262234031s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.392321345 (583h22m23.26340438s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:23.394079863 (583h22m23.265162908s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Uploading piece... Dec 29 22:51:25.082580799 (583h22m24.953663834s) [upload=511377]: Next returned nil segment Dec 29 22:51:25.082623130 (583h22m24.953706165s) [upload=511377]: Waiting for error group managing segments... Dec 29 22:51:25.366111798 (583h22m25.237194833s) [noise=true , address=145.79.227.45:4231 , piece_id=HX2CRXSOINRCP7P3VMMGKPVZSF7BHLV3QPBGAQQ3CZYQRMQ5GEPQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.367107353 (583h22m25.238190388s) [noise=true , address=145.79.227.5:4221 , piece_id=RPQZOKTJUHSMHONJIQLSQK5DKFX66WOPFR5DEOICYUSL6BHNEQ7Q, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.375498331 (583h22m25.246581376s) [noise=true , address=145.79.227.46:4601 , piece_id=GHVWKDICVGWXOJDC6TTV7DAAMBJTJHPKZQNAKEA654ZWKWCQ6ANQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.377770518 (583h22m25.248853553s) [noise=true , address=145.79.227.15:4873 , piece_id=HSPMLGINVXHSLTWP36EWJ347QDYWZS3CODYEWY2SOWUMX7BB7YFQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.381116128 (583h22m25.252199173s) [noise=true , address=145.79.227.2:4115 , piece_id=EB42TKBQ7MEP5P7Y2EZI2UBTKOFPZG7Q43TYJFFT6IJSO7QYUUBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.384339525 (583h22m25.25542256s) [noise=true , address=145.79.227.31:4797 , piece_id=NXFTQSZVFXOWD6FWRRLYIX7MGMRGEJBWCQSLAQSIFT6GLTGTFKSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.394437774 (583h22m25.265520809s) [noise=true , address=145.79.227.52:4445 , piece_id=JF7YOKAOW55KVX3PHPF53BDWX4PLYYZJDW5D5PMYK3GWZYPIDBKA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.401508969 (583h22m25.272592014s) [noise=true , address=145.79.227.22:4509 , piece_id=MBTYR77LTA7QCOYIWOYL4PFLPQO7VAA35ZRGWJXHNJVSAI6ACGSQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.409500305 (583h22m25.28058334s) [noise=true , address=145.79.227.48:4302 , piece_id=H2I4QGTT5X6MHN2LX3BRCD4WOGOXLQ4S42OXCMNY63Y5SH62S72A, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.551269461 (583h22m25.422352507s) [noise=true , address=77.90.31.14:4664 , piece_id=5ICZFTZ7ST7Z3BFIOKYTIW7F7ZXMW4DQBCVN4JVWQJNZUJ3QQGHA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.577925347 (583h22m25.449008392s) [noise=true , address=77.90.31.10:4520 , piece_id=MNJ3LUSVPY5UONZMGY6EMDVY475K6F6GRH7IUCI44QJ26EC3W6UA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.654794526 (583h22m25.525877561s) [noise=true , address=45.91.116.5:4297 , piece_id=PNJQ6N6VUQVYXYYHRCTA6UHK3S5PT42KNLXUYYPJ3GG3OR7JBSDA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685418388 (583h22m25.556501453s) [noise=true , address=45.91.116.9:4263 , piece_id=E5TIUWSUOE6U7FJPYEI5KS5MHFRCRVWXBJJQY4A4JE53QILOUWBQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: <nil> Dec 29 22:51:25.685445069 (583h22m25.556528104s) [seg_pos={0 0}, upload=511377]: Segment reached optimal threshold of 13 pieces. Dec 29 22:51:25.685679314 (583h22m25.556762359s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685688863 (583h22m25.556771908s) [noise=true , address=45.91.116.8:4046 , piece_id=3YXRM44QQGBXJ32AXV5XXJ6PDIOQS5R2JMB5QEH3B2FBFWHCNLAA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685714000 (583h22m25.556797035s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685724801 (583h22m25.556807846s) [noise=true , address=77.90.31.21:4880 , piece_id=G2C7DIBBDTF5UI6J3EU7QVNJIKPQMR6GW2RWQRZ7W2VZ7D5KHVMQ, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685930592 (583h22m25.557013627s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 22:51:25.685942846 (583h22m25.557025911s) [noise=true , address=45.91.116.6:4973 , piece_id=RP6HG5Z3SN4273XE4FFSVUZBWE2RFA3YBZ3O2QDGMVALHKTQBFJA, seg_pos={0 0}, upload=511377]: Long tail cancellation due to optimal threshold Dec 29 22:51:25.685982932 (583h22m25.557065967s) [seg_pos={0 0}, upload=511377]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:51:25.686093312 (583h22m25.557176357s) [upload=511377]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:51:25.686102319 (583h22m25.557185364s) [upload=511377]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:51:25.970728401 (583h22m25.841811436s) [upload=511377]: Done uploading segments... Dec 29 22:51:25.970748099 (583h22m25.841831134s) [upload=511377]: Upload finished. err: <nil> Duration: 30.00263091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..................X......X.............................] duration: 20.119613334s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:51:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 20.119639845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.............X......X...............] duration: 5.975129633s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:51:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 5.975158358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..................X......X............................] duration: 15.446218788s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:50:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.446244347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.............X......X..............] duration: 6.094863483s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.094892999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..................X......X...........................] duration: 16.934681136s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 16.934717274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.............X......X.............] duration: 6.103621487s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:49:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.103650464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..................X......X..........................] duration: 16.138008568s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 16.13803592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........................X.............X......X............] duration: 5.838496157s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 5.838523179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..................X......X.........................] duration: 15.317294423s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.317321023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..................X......X........................] duration: 26.130049188s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 26.130084123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........................X.............X......X...........] duration: 6.061742273s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:47:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.061761839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..................X......X.......................] duration: 24.596292973s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 24.596312862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........................X.............X......X..........] duration: 6.204778262s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:46:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.204817365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........................X.............X......X.........] duration: 20.935093541s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 20.935138707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..................X......X......................] duration: 27.978868016s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 27.978895409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........................X.............X......X........] duration: 7.243088643s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 7.243114571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..................X......X.....................] duration: 15.440481705s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.440503466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........................X.............X......X.......] duration: 6.168102902s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.168127518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.........................X.............X......X......] duration: 7.212354238s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 7.21238138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..................X......X....................] duration: 15.349779351s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:44:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 15.349796244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..................X......X...................] duration: 25.325874954s trace: https://jaeger.dev.storj.io/trace/282f8c6bbcce60e6 FAIL (2025-12-29 22:43:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, Piece ID: JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA) context deadline exceeded; piecestore: (Node ID: 12EetZrv8PDwKiXhUS1aKEMHM1MPDpVL6NygzfLskY9dFVJN9yi, Piece ID: KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA) context deadline exceeded; piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A) context deadline exceeded ======LOGS====== 94 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379243801 (582h59m30.250326836s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379375892 (582h59m30.250458927s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379643001 (582h59m30.250726036s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379810780 (582h59m30.250893815s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.379955225 (582h59m30.25103826s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380200131 (582h59m30.251283166s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380240889 (582h59m30.251323934s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380514821 (582h59m30.251597866s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380536182 (582h59m30.251619217s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380769064 (582h59m30.251852099s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.380948265 (582h59m30.252031311s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381305126 (582h59m30.252388161s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.381846867 (582h59m30.252929902s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:30.382290792 (582h59m30.253373827s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Uploading piece... Dec 29 22:28:31.685069707 (582h59m31.556152752s) [upload=511045]: Next returned nil segment Dec 29 22:28:31.685092309 (582h59m31.556175354s) [upload=511045]: Waiting for error group managing segments... Dec 29 22:28:32.026984168 (582h59m31.898067203s) [noise=true , address=64.49.13.140:4511 , piece_id=5HFHF6YFQBLFIWN5TX26UZHSPXGHJ5ABSST6BLNAM2DXZFPNQEQA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.274464964 (582h59m32.145547999s) [noise=true , address=145.79.227.25:4626 , piece_id=32AISH5BHLVBEMGHX6XS5M6ALGYDFACWYOVFDZG5MYSQFFMNRC5Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.287552263 (582h59m32.158635318s) [noise=true , address=5.231.73.2:4018 , piece_id=KMJCWKRAZA6OANZOTUXLERKGASGNVUO7ZCBUNUNLYBNRLTBABEVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.291740760 (582h59m32.162823805s) [noise=true , address=145.79.227.16:4915 , piece_id=E6H3NZXZ2NSBF7L7JHZO3MCRM5MDH6FF5EQO627YPI5WHTFFA32A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.425447855 (582h59m32.29653089s) [noise=true , address=77.90.31.12:4619 , piece_id=7FDIQ7L2DQHYDJ67SHZLWH3ZMTOF53QUZHK64FCIJARSJAVEOKPQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.446860348 (582h59m32.317943383s) [noise=true , address=77.90.31.21:4882 , piece_id=JQPG4A2KIUGGQ5X63A5X4G52HJTNRCME6PZFCZAQ6MDBP7URIBGA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:32.467889360 (582h59m32.338972395s) [noise=true , address=77.90.31.16:4762 , piece_id=ABK6B2LEJYW5VLZXMB2Q4CPCFR5UFYFIVDEEMJHTULEQHW6M5UZQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.096259409 (582h59m32.967342454s) [noise=true , address=77.90.31.18:4819 , piece_id=BLHXQVOWKQJVRCPM4JHLXPXHKULXAAX7BTJDWN3E2A2QFLKGLT3A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.135405278 (582h59m33.006488313s) [noise=true , address=145.79.227.43:4171 , piece_id=6PHAG6UE43U6MFSS3VRV33KLX2CBVLWSGFEPTPRINFMK6I2PHQPA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.153401575 (582h59m33.02448461s) [noise=true , address=145.79.227.40:4058 , piece_id=72JU7WCY4CY3S2OAF4R2KYVHV5VAMYISP2F5WRPWMTGYMUK6DZVA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.172428365 (582h59m33.04351141s) [noise=true , address=145.79.227.35:4501 , piece_id=3EEORLOYUBKKLXYYN25Z6MPJOSSMR4TDP7NEY3C74J6TJYVEC52Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.197928761 (582h59m33.069011806s) [noise=true , address=77.90.31.10:4534 , piece_id=43M4GBB2OHIGCFVRALJ6RTYEJLR4VMLMWBJRUYDJA5XBHZEDW64A, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.208021328 (582h59m33.079104373s) [noise=true , address=145.79.227.30:4766 , piece_id=TZIBGQ4VD7TMS6G3HGRAONU5CER6BD6EZFRDJXJEGWKSZCIS5M7Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210946779 (582h59m33.082029814s) [noise=true , address=145.79.227.34:4939 , piece_id=JZKKNUGY3SUU7Y6X5FQF2YAY6QZ3XBVVL2SRUFJNJ45JM5GATMOQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: <nil> Dec 29 22:28:33.210957459 (582h59m33.082040494s) [seg_pos={0 0}, upload=511045]: Segment reached optimal threshold of 14 pieces. Dec 29 22:28:33.211170935 (582h59m33.08225398s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60418->64.49.13.169:4709: use of closed network connection Dec 29 22:28:33.211180614 (582h59m33.082263659s) [noise=true , address=64.49.13.169:4709 , piece_id=IGZICZ42VQGYPGSKOEULMYVFHGQDJYED3D56LA4TBCTMC4SQX45Q, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211556589 (582h59m33.082639624s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211565988 (582h59m33.082649023s) [noise=true , address=145.79.227.44:4223 , piece_id=BDEBPMOYABILWORS57Y7K46D4GKES3DWMHO2YNB6SAKE7JPZB5IQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211720081 (582h59m33.082803126s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz): protocol: expected piece hash; context canceled; EOF Dec 29 22:28:33.211728336 (582h59m33.082811372s) [noise=true , address=145.79.227.30:4783 , piece_id=KGXY4KIMQ2322YUWEF4TIF7TI4PZ4OYQMOH5S77QPAHMXEM6RRLQ, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211916545 (582h59m33.082999591s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf): protocol: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54972->145.79.227.24:4594: use of closed network connection Dec 29 22:28:33.211922968 (582h59m33.083006003s) [noise=true , address=145.79.227.24:4594 , piece_id=QU2RSB3NP7NB2A4Z6LBAZPKHSUV6OAZQNFS6TO2PALRDI45M7EOA, seg_pos={0 0}, upload=511045]: Long tail cancellation due to optimal threshold Dec 29 22:28:33.211940861 (582h59m33.083023896s) [seg_pos={0 0}, upload=511045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:28:33.212005124 (582h59m33.083088159s) [upload=511045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:28:33.212009251 (582h59m33.083092296s) [upload=511045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:28:33.526396404 (582h59m33.39747944s) [upload=511045]: Done uploading segments... Dec 29 22:28:33.526415111 (582h59m33.397498156s) [upload=511045]: Upload finished. err: <nil> Duration: 25.32589361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.........................X.............X......X.....] duration: 6.321415858s trace: https://jaeger.dev.storj.io/trace/4431034e00544096 FAIL (2025-12-29 22:43:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 22:40:11.472147401 (583h11m11.343230446s) [upload=511211]: Got next segment. Inline: false Dec 29 22:40:11.472215059 (583h11m11.343298095s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:40:11.472222304 (583h11m11.343305349s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:40:11.721049189 (583h11m11.592132235s) [seg_pos={0 0}, upload=511211]: Begin upload segment... Dec 29 22:40:11.721120375 (583h11m11.59220341s) [seg_pos={0 0}, upload=511211]: Done begin upload segment. Dec 29 22:40:11.721282664 (583h11m11.592365709s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.721346426 (583h11m11.592429471s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722125619 (583h11m11.593208664s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.722893351 (583h11m11.593976396s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723478576 (583h11m11.594561621s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.723867686 (583h11m11.594950731s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724250796 (583h11m11.595333841s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724567679 (583h11m11.595650714s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.724854295 (583h11m11.59593734s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725225191 (583h11m11.596308236s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.725606558 (583h11m11.596689603s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.726178025 (583h11m11.59726106s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.727231882 (583h11m11.598314927s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728272453 (583h11m11.599355498s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728353698 (583h11m11.599436743s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:11.728813884 (583h11m11.599896919s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Uploading piece... Dec 29 22:40:13.423196279 (583h11m13.294279314s) [upload=511211]: Next returned nil segment Dec 29 22:40:13.423228320 (583h11m13.294311366s) [upload=511211]: Waiting for error group managing segments... Dec 29 22:40:13.702882855 (583h11m13.5739659s) [noise=true , address=145.79.227.19:4077 , piece_id=OOMRX6GJUHGXVXHJ5NWI6HTYV7JAOMZHUOAYYIGBUN4WFJH2O3IA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.703135546 (583h11m13.574218591s) [noise=true , address=145.79.227.22:4509 , piece_id=XP5KHKY6CQ5FCPLSR6KX22D6ZF6DDSGAKER2Q73LYYR7A44LNIUQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.707362635 (583h11m13.57844568s) [noise=true , address=145.79.227.36:4518 , piece_id=47ZGADTF5HAT67ZEU4W4CBFVVVYPXPOSABQAEWWH4CZN5ORUBVMQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.723036350 (583h11m13.594119395s) [noise=true , address=145.79.227.17:4972 , piece_id=D3GK66PJW2RUPNUIY22ECWRZ4YYTINMAS25XNBF57NMUYDODKJVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.729927152 (583h11m13.601010187s) [noise=true , address=145.79.227.21:4475 , piece_id=RLURLHVCZ7OZECIWELZ7T5D3KVYMKIFLUEPFKRCY7J6UDXT5IKPQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.737656008 (583h11m13.608739053s) [noise=true , address=145.79.227.50:4375 , piece_id=VXXDP6O6YAODSA7TD74H3QY7YRNPBX7JI2TNNCRFICRJMOVG67UA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.748525535 (583h11m13.61960857s) [noise=true , address=145.79.227.47:4267 , piece_id=7C4OZH7DRQYAORILEW2VWENUSKLPRDU6EKEPYWCB5CG5RLPCRPTA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.888628427 (583h11m13.759711462s) [noise=true , address=77.90.31.10:4520 , piece_id=OIM2Y77LF7G62BX2GUVUUE5P5CTTE67HHWTKKHVZWKZ2T5QLDHDA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.901212930 (583h11m13.772295975s) [noise=true , address=77.90.31.19:4844 , piece_id=3XKJ3BJOPR7Z5PWRZSNOHLTONBQZUAAYJPOVVJXR7SQH6DJKFXJQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:13.989380252 (583h11m13.860463287s) [noise=true , address=45.91.116.9:4262 , piece_id=SWCNRI7623GLMSAN4PDDB7Z2HV7FXDFCCOLL2FRTLNTPNX3NY63A, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.031710673 (583h11m13.902793708s) [noise=true , address=45.91.116.7:4011 , piece_id=FFFHZGBXGZPCZL2UZRDQTC4Q7X37AEHUGECF7OGILMHPYB3ICKEA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.100675140 (583h11m13.971758175s) [noise=true , address=45.91.116.5:4299 , piece_id=JFZZDOKGMDL4G3BQ6XBE6X6NAVIVZIEZZT5KDDF2BBPWD3CPDLHQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979934907 (583h11m14.851017962s) [noise=true , address=145.79.227.29:4942 , piece_id=I5RP4UGINHPQ5MAA2JG7HKQS2WSF7VIZQOGKZMK7KJCL5CAA24TA, seg_pos={0 0}, upload=511211]: Done uploading piece. err: <nil> Dec 29 22:40:14.979967038 (583h11m14.851050083s) [seg_pos={0 0}, upload=511211]: Segment reached optimal threshold of 13 pieces. Dec 29 22:40:14.980218296 (583h11m14.851301341s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 22:40:14.980225280 (583h11m14.851308315s) [noise=true , address=145.79.227.43:4159 , piece_id=CVFVTKXSNQ576C7CKWZSH33AXQT4PVW65ZC5YTC3OT3YEJOJVZAQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980317826 (583h11m14.851400861s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55096->45.91.116.6:4974: use of closed network connection Dec 29 22:40:14.980342192 (583h11m14.851425227s) [noise=true , address=45.91.116.6:4974 , piece_id=7BO53ADCWJOTKFN6AFTBZZTIXDEU4A73BNFL2L3562M7SXO4L4KQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980473182 (583h11m14.851556227s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33462->145.79.227.41:4086: use of closed network connection Dec 29 22:40:14.980480265 (583h11m14.8515633s) [noise=true , address=145.79.227.41:4086 , piece_id=WU7GUOSLN2GZOBJAL67E4JAE3BGCO2OXXEYJ4ANNV3EYDYMLAYVQ, seg_pos={0 0}, upload=511211]: Long tail cancellation due to optimal threshold Dec 29 22:40:14.980500735 (583h11m14.85158377s) [seg_pos={0 0}, upload=511211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:40:14.980575337 (583h11m14.851658382s) [upload=511211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:40:14.980579965 (583h11m14.851663s) [upload=511211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:40:15.281623138 (583h11m15.152706213s) [upload=511211]: Done uploading segments... Dec 29 22:40:15.281649227 (583h11m15.152732273s) [upload=511211]: Upload finished. err: <nil> Duration: 6.321502423s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)