Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [...XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.] duration: 5.001009342s trace: https://jaeger.dev.storj.io/trace/d69e22cdb3422c1 FAIL (2025-12-28 05:41:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001038617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X....] duration: 15.190574284s trace: https://jaeger.dev.storj.io/trace/28251b7f99b526b4 FAIL (2025-12-28 05:41:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q) context deadline exceeded ======LOGS====== 42]: Starting upload Dec 28 05:38:47.679321549 (542h9m47.550404624s) [upload=475442]: Done starting upload Dec 28 05:38:47.679443661 (542h9m47.550526696s) [upload=475442]: Uploading segments... Dec 28 05:38:47.679553170 (542h9m47.550636205s) [upload=475442]: Got next segment. Inline: false Dec 28 05:38:47.679598386 (542h9m47.550681441s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:38:47.679605209 (542h9m47.550688244s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:38:47.924329113 (542h9m47.795412148s) [seg_pos={0 0}, upload=475442]: Begin upload segment... Dec 28 05:38:47.924416468 (542h9m47.795499514s) [seg_pos={0 0}, upload=475442]: Done begin upload segment. Dec 28 05:38:47.924479889 (542h9m47.795562934s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924493044 (542h9m47.795576079s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924588947 (542h9m47.795671982s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924915759 (542h9m47.795998794s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925237562 (542h9m47.796320607s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925574954 (542h9m47.796657989s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925959717 (542h9m47.797042752s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926316617 (542h9m47.797399662s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926965181 (542h9m47.798048227s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.928429269 (542h9m47.799512304s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.929862008 (542h9m47.800945043s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932025688 (542h9m47.803108723s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932247540 (542h9m47.803330575s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.938337537 (542h9m47.809420572s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.940721606 (542h9m47.811804641s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.942185104 (542h9m47.813268169s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:49.637502550 (542h9m49.508585605s) [upload=475442]: Next returned nil segment Dec 28 05:38:49.637555791 (542h9m49.508638826s) [upload=475442]: Waiting for error group managing segments... Dec 28 05:38:49.926862707 (542h9m49.797945742s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.931341165 (542h9m49.8024242s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.942675989 (542h9m49.813759024s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.952101937 (542h9m49.823184972s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.958348691 (542h9m49.829431736s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.959733308 (542h9m49.830816343s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.961543184 (542h9m49.832626219s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.964195665 (542h9m49.8352787s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.171018760 (542h9m50.042101805s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.191800280 (542h9m50.062883325s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.234919806 (542h9m50.106002852s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.236447826 (542h9m50.107530861s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250044186 (542h9m50.121127231s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250071778 (542h9m50.121154823s) [seg_pos={0 0}, upload=475442]: Segment reached optimal threshold of 13 pieces. Dec 28 05:38:50.250392940 (542h9m50.121475976s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection Dec 28 05:38:50.250407337 (542h9m50.121490373s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250421324 (542h9m50.121504389s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:44680->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:38:50.250432756 (542h9m50.121515801s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250787111 (542h9m50.121870146s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:38:50.250795817 (542h9m50.121878852s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250817609 (542h9m50.121900644s) [seg_pos={0 0}, upload=475442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:38:50.250899995 (542h9m50.121983061s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:38:50.250908010 (542h9m50.121991056s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:38:50.535462890 (542h9m50.406545975s) [upload=475442]: Done uploading segments... Dec 28 05:38:50.535489710 (542h9m50.406572755s) [upload=475442]: Upload finished. err: <nil> Duration: 15.190601846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............................X.........X...X...........] duration: 45.151731846s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:41:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 45.151753146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [....XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX] duration: 5.000572872s trace: https://jaeger.dev.storj.io/trace/d69e22cdb3422c1 FAIL (2025-12-28 05:41:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000916406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X...] duration: 5.976322373s trace: https://jaeger.dev.storj.io/trace/28251b7f99b526b4 FAIL (2025-12-28 05:41:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q) context deadline exceeded ======LOGS====== 42]: Starting upload Dec 28 05:38:47.679321549 (542h9m47.550404624s) [upload=475442]: Done starting upload Dec 28 05:38:47.679443661 (542h9m47.550526696s) [upload=475442]: Uploading segments... Dec 28 05:38:47.679553170 (542h9m47.550636205s) [upload=475442]: Got next segment. Inline: false Dec 28 05:38:47.679598386 (542h9m47.550681441s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:38:47.679605209 (542h9m47.550688244s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:38:47.924329113 (542h9m47.795412148s) [seg_pos={0 0}, upload=475442]: Begin upload segment... Dec 28 05:38:47.924416468 (542h9m47.795499514s) [seg_pos={0 0}, upload=475442]: Done begin upload segment. Dec 28 05:38:47.924479889 (542h9m47.795562934s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924493044 (542h9m47.795576079s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924588947 (542h9m47.795671982s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924915759 (542h9m47.795998794s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925237562 (542h9m47.796320607s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925574954 (542h9m47.796657989s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925959717 (542h9m47.797042752s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926316617 (542h9m47.797399662s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926965181 (542h9m47.798048227s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.928429269 (542h9m47.799512304s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.929862008 (542h9m47.800945043s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932025688 (542h9m47.803108723s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932247540 (542h9m47.803330575s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.938337537 (542h9m47.809420572s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.940721606 (542h9m47.811804641s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.942185104 (542h9m47.813268169s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:49.637502550 (542h9m49.508585605s) [upload=475442]: Next returned nil segment Dec 28 05:38:49.637555791 (542h9m49.508638826s) [upload=475442]: Waiting for error group managing segments... Dec 28 05:38:49.926862707 (542h9m49.797945742s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.931341165 (542h9m49.8024242s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.942675989 (542h9m49.813759024s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.952101937 (542h9m49.823184972s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.958348691 (542h9m49.829431736s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.959733308 (542h9m49.830816343s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.961543184 (542h9m49.832626219s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.964195665 (542h9m49.8352787s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.171018760 (542h9m50.042101805s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.191800280 (542h9m50.062883325s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.234919806 (542h9m50.106002852s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.236447826 (542h9m50.107530861s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250044186 (542h9m50.121127231s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250071778 (542h9m50.121154823s) [seg_pos={0 0}, upload=475442]: Segment reached optimal threshold of 13 pieces. Dec 28 05:38:50.250392940 (542h9m50.121475976s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection Dec 28 05:38:50.250407337 (542h9m50.121490373s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250421324 (542h9m50.121504389s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:44680->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:38:50.250432756 (542h9m50.121515801s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250787111 (542h9m50.121870146s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:38:50.250795817 (542h9m50.121878852s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250817609 (542h9m50.121900644s) [seg_pos={0 0}, upload=475442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:38:50.250899995 (542h9m50.121983061s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:38:50.250908010 (542h9m50.121991056s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:38:50.535462890 (542h9m50.406545975s) [upload=475442]: Done uploading segments... Dec 28 05:38:50.535489710 (542h9m50.406572755s) [upload=475442]: Upload finished. err: <nil> Duration: 5.976354083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............................X.........X...X..........] duration: 41.244802007s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:41:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 41.244829209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.....XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...X] duration: 5.002076785s trace: https://jaeger.dev.storj.io/trace/58a03e9bda1a04d9 FAIL (2025-12-28 05:40:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002105709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............................X.........X...X.........] duration: 41.221883737s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 41.221915708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X..] duration: 5.898233071s trace: https://jaeger.dev.storj.io/trace/28251b7f99b526b4 FAIL (2025-12-28 05:39:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q) context deadline exceeded ======LOGS====== 42]: Starting upload Dec 28 05:38:47.679321549 (542h9m47.550404624s) [upload=475442]: Done starting upload Dec 28 05:38:47.679443661 (542h9m47.550526696s) [upload=475442]: Uploading segments... Dec 28 05:38:47.679553170 (542h9m47.550636205s) [upload=475442]: Got next segment. Inline: false Dec 28 05:38:47.679598386 (542h9m47.550681441s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:38:47.679605209 (542h9m47.550688244s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:38:47.924329113 (542h9m47.795412148s) [seg_pos={0 0}, upload=475442]: Begin upload segment... Dec 28 05:38:47.924416468 (542h9m47.795499514s) [seg_pos={0 0}, upload=475442]: Done begin upload segment. Dec 28 05:38:47.924479889 (542h9m47.795562934s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924493044 (542h9m47.795576079s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924588947 (542h9m47.795671982s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924915759 (542h9m47.795998794s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925237562 (542h9m47.796320607s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925574954 (542h9m47.796657989s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925959717 (542h9m47.797042752s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926316617 (542h9m47.797399662s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926965181 (542h9m47.798048227s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.928429269 (542h9m47.799512304s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.929862008 (542h9m47.800945043s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932025688 (542h9m47.803108723s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932247540 (542h9m47.803330575s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.938337537 (542h9m47.809420572s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.940721606 (542h9m47.811804641s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.942185104 (542h9m47.813268169s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:49.637502550 (542h9m49.508585605s) [upload=475442]: Next returned nil segment Dec 28 05:38:49.637555791 (542h9m49.508638826s) [upload=475442]: Waiting for error group managing segments... Dec 28 05:38:49.926862707 (542h9m49.797945742s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.931341165 (542h9m49.8024242s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.942675989 (542h9m49.813759024s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.952101937 (542h9m49.823184972s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.958348691 (542h9m49.829431736s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.959733308 (542h9m49.830816343s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.961543184 (542h9m49.832626219s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.964195665 (542h9m49.8352787s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.171018760 (542h9m50.042101805s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.191800280 (542h9m50.062883325s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.234919806 (542h9m50.106002852s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.236447826 (542h9m50.107530861s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250044186 (542h9m50.121127231s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250071778 (542h9m50.121154823s) [seg_pos={0 0}, upload=475442]: Segment reached optimal threshold of 13 pieces. Dec 28 05:38:50.250392940 (542h9m50.121475976s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection Dec 28 05:38:50.250407337 (542h9m50.121490373s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250421324 (542h9m50.121504389s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:44680->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:38:50.250432756 (542h9m50.121515801s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250787111 (542h9m50.121870146s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:38:50.250795817 (542h9m50.121878852s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250817609 (542h9m50.121900644s) [seg_pos={0 0}, upload=475442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:38:50.250899995 (542h9m50.121983061s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:38:50.250908010 (542h9m50.121991056s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:38:50.535462890 (542h9m50.406545975s) [upload=475442]: Done uploading segments... Dec 28 05:38:50.535489710 (542h9m50.406572755s) [upload=475442]: Upload finished. err: <nil> Duration: 5.898256845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.] duration: 5.763131727s trace: https://jaeger.dev.storj.io/trace/28251b7f99b526b4 FAIL (2025-12-28 05:39:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q) context deadline exceeded ======LOGS====== 42]: Starting upload Dec 28 05:38:47.679321549 (542h9m47.550404624s) [upload=475442]: Done starting upload Dec 28 05:38:47.679443661 (542h9m47.550526696s) [upload=475442]: Uploading segments... Dec 28 05:38:47.679553170 (542h9m47.550636205s) [upload=475442]: Got next segment. Inline: false Dec 28 05:38:47.679598386 (542h9m47.550681441s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:38:47.679605209 (542h9m47.550688244s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:38:47.924329113 (542h9m47.795412148s) [seg_pos={0 0}, upload=475442]: Begin upload segment... Dec 28 05:38:47.924416468 (542h9m47.795499514s) [seg_pos={0 0}, upload=475442]: Done begin upload segment. Dec 28 05:38:47.924479889 (542h9m47.795562934s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924493044 (542h9m47.795576079s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924588947 (542h9m47.795671982s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924915759 (542h9m47.795998794s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925237562 (542h9m47.796320607s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925574954 (542h9m47.796657989s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925959717 (542h9m47.797042752s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926316617 (542h9m47.797399662s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926965181 (542h9m47.798048227s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.928429269 (542h9m47.799512304s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.929862008 (542h9m47.800945043s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932025688 (542h9m47.803108723s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932247540 (542h9m47.803330575s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.938337537 (542h9m47.809420572s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.940721606 (542h9m47.811804641s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.942185104 (542h9m47.813268169s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:49.637502550 (542h9m49.508585605s) [upload=475442]: Next returned nil segment Dec 28 05:38:49.637555791 (542h9m49.508638826s) [upload=475442]: Waiting for error group managing segments... Dec 28 05:38:49.926862707 (542h9m49.797945742s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.931341165 (542h9m49.8024242s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.942675989 (542h9m49.813759024s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.952101937 (542h9m49.823184972s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.958348691 (542h9m49.829431736s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.959733308 (542h9m49.830816343s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.961543184 (542h9m49.832626219s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.964195665 (542h9m49.8352787s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.171018760 (542h9m50.042101805s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.191800280 (542h9m50.062883325s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.234919806 (542h9m50.106002852s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.236447826 (542h9m50.107530861s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250044186 (542h9m50.121127231s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250071778 (542h9m50.121154823s) [seg_pos={0 0}, upload=475442]: Segment reached optimal threshold of 13 pieces. Dec 28 05:38:50.250392940 (542h9m50.121475976s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection Dec 28 05:38:50.250407337 (542h9m50.121490373s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250421324 (542h9m50.121504389s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:44680->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:38:50.250432756 (542h9m50.121515801s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250787111 (542h9m50.121870146s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:38:50.250795817 (542h9m50.121878852s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250817609 (542h9m50.121900644s) [seg_pos={0 0}, upload=475442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:38:50.250899995 (542h9m50.121983061s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:38:50.250908010 (542h9m50.121991056s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:38:50.535462890 (542h9m50.406545975s) [upload=475442]: Done uploading segments... Dec 28 05:38:50.535489710 (542h9m50.406572755s) [upload=475442]: Upload finished. err: <nil> Duration: 5.76318063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...] duration: 5.001719244s trace: https://jaeger.dev.storj.io/trace/768e65a7989ace8 FAIL (2025-12-28 05:39:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001745115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X] duration: 30.002253815s trace: https://jaeger.dev.storj.io/trace/28251b7f99b526b4 FAIL (2025-12-28 05:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q) context deadline exceeded ======LOGS====== 42]: Starting upload Dec 28 05:38:47.679321549 (542h9m47.550404624s) [upload=475442]: Done starting upload Dec 28 05:38:47.679443661 (542h9m47.550526696s) [upload=475442]: Uploading segments... Dec 28 05:38:47.679553170 (542h9m47.550636205s) [upload=475442]: Got next segment. Inline: false Dec 28 05:38:47.679598386 (542h9m47.550681441s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:38:47.679605209 (542h9m47.550688244s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:38:47.924329113 (542h9m47.795412148s) [seg_pos={0 0}, upload=475442]: Begin upload segment... Dec 28 05:38:47.924416468 (542h9m47.795499514s) [seg_pos={0 0}, upload=475442]: Done begin upload segment. Dec 28 05:38:47.924479889 (542h9m47.795562934s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924493044 (542h9m47.795576079s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924588947 (542h9m47.795671982s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.924915759 (542h9m47.795998794s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925237562 (542h9m47.796320607s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925574954 (542h9m47.796657989s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.925959717 (542h9m47.797042752s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926316617 (542h9m47.797399662s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.926965181 (542h9m47.798048227s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.928429269 (542h9m47.799512304s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.929862008 (542h9m47.800945043s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932025688 (542h9m47.803108723s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.932247540 (542h9m47.803330575s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.938337537 (542h9m47.809420572s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.940721606 (542h9m47.811804641s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:47.942185104 (542h9m47.813268169s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Uploading piece... Dec 28 05:38:49.637502550 (542h9m49.508585605s) [upload=475442]: Next returned nil segment Dec 28 05:38:49.637555791 (542h9m49.508638826s) [upload=475442]: Waiting for error group managing segments... Dec 28 05:38:49.926862707 (542h9m49.797945742s) [noise=true , address=145.79.227.36:4519 , piece_id=M4UZ4HNY73DCRNTXE6CALKEB7YS3AU56A5PL33BE52GT4RXQCJIQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.931341165 (542h9m49.8024242s) [noise=true , address=145.79.227.6:4257 , piece_id=VF3GYNIQNMRMVYDNRZRZXRPHR7OIF2U7ZCKAF7EL755FRI2SR3SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.942675989 (542h9m49.813759024s) [noise=true , address=145.79.227.10:4403 , piece_id=MPL4SCQUYOYETH5PWGY3FSBRSS3UXBB6DZ3PEJSHGQ2ZHWPT6SDA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.952101937 (542h9m49.823184972s) [noise=true , address=145.79.227.27:4691 , piece_id=VSUXCZOGUXJTIULWYRSY2UOJL2UBDHHTU3BJLVGSLHX4IU5TSASA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.958348691 (542h9m49.829431736s) [noise=true , address=145.79.227.48:4302 , piece_id=UW5BLD7TGG7F2WKBAEVAJZBQODATVJKUSWUQUA3M3NAPKOEP5SPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.959733308 (542h9m49.830816343s) [noise=true , address=145.79.227.50:4374 , piece_id=PGKZNHNZFCTKTCQP23VYMLIOAVXBZXII2V2BDUZ3FX3FI6GX2VWA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.961543184 (542h9m49.832626219s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZCW2OHFN4LNWREV5NRDWBX6X73VYHCZ6O54A7PPANLABX3AP4SA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:49.964195665 (542h9m49.8352787s) [noise=true , address=145.79.227.31:4797 , piece_id=2ZD3J734OJHYPSZLWWFDJHTSETVA3B3M4H2N5WFJEUFX4THQSSTQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.171018760 (542h9m50.042101805s) [noise=true , address=45.91.116.5:4298 , piece_id=M5FZYWHBZDVE3M23QWAEX5Z5VOUJ32N2EFWOHU5LWBCKAP2CRBOA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.191800280 (542h9m50.062883325s) [noise=true , address=45.91.116.11:4118 , piece_id=OOUSCI5WYL5N6FKARGHMBRP7ANDZIEBWWH5NSR3JZKJLVRSQ3J5Q, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.234919806 (542h9m50.106002852s) [noise=true , address=45.91.116.12:4228 , piece_id=RBURZUDMWALWMLIWYBDRTDAAZ63QNDKIWIZYNW2JBI4LZBFMCUHQ, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.236447826 (542h9m50.107530861s) [noise=true , address=45.91.116.8:4048 , piece_id=6OEYD6XOJTSL4DV2Q7RJKZ3LBLLL3F4EWZ2D3ZKLBAWXZZYDZJEA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250044186 (542h9m50.121127231s) [noise=true , address=77.90.31.10:4519 , piece_id=CPCZLKEXGEEJ4ADTWXPYZEX7LT2BK3K35IYFHH7Z5ST2LKGXP6GA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: <nil> Dec 28 05:38:50.250071778 (542h9m50.121154823s) [seg_pos={0 0}, upload=475442]: Segment reached optimal threshold of 13 pieces. Dec 28 05:38:50.250392940 (542h9m50.121475976s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36466->45.91.116.13:4154: use of closed network connection Dec 28 05:38:50.250407337 (542h9m50.121490373s) [noise=true , address=45.91.116.13:4154 , piece_id=B7XKJHTBCCZCTMP3GB3GZ4PUEIB3HDNCL2CG6HOVD3D4HUW4GBPA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250421324 (542h9m50.121504389s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:44680->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:38:50.250432756 (542h9m50.121515801s) [noise=true , address=45.91.116.6:4975 , piece_id=3KXHXDSUNO3AYVWHFCRRPARLEOGKW2SM7CYCUX46K3SRTLGWHLGA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250787111 (542h9m50.121870146s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:38:50.250795817 (542h9m50.121878852s) [noise=true , address=45.91.116.7:4011 , piece_id=XDJ2ZHOQPV2P5SI7O2A7SA3QX7XFRQCXALLLNJQN324ABXCTZGTA, seg_pos={0 0}, upload=475442]: Long tail cancellation due to optimal threshold Dec 28 05:38:50.250817609 (542h9m50.121900644s) [seg_pos={0 0}, upload=475442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:38:50.250899995 (542h9m50.121983061s) [upload=475442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:38:50.250908010 (542h9m50.121991056s) [upload=475442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:38:50.535462890 (542h9m50.406545975s) [upload=475442]: Done uploading segments... Dec 28 05:38:50.535489710 (542h9m50.406572755s) [upload=475442]: Upload finished. err: <nil> Duration: 30.002276899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............................X.........X...X........] duration: 55.419200186s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:39:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 55.41921783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X..] duration: 5.001045833s trace: https://jaeger.dev.storj.io/trace/768e65a7989ace8 FAIL (2025-12-28 05:39:06 UTC) orbiter warning: context deadline exceeded Duration: 5.001079938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X.] duration: 4.958621179s trace: https://jaeger.dev.storj.io/trace/768e65a7989ace8 FAIL (2025-12-28 05:38:49 UTC) orbiter warning: context deadline exceeded Duration: 4.958649392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..] duration: 5.771062177s trace: https://jaeger.dev.storj.io/trace/505bab7549ef4ee6 FAIL (2025-12-28 05:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A) context deadline exceeded ======LOGS====== Dec 28 05:37:26.527890652 (542h8m26.398973707s) [upload=475418]: Starting upload Dec 28 05:37:26.527974341 (542h8m26.399057386s) [upload=475418]: Done starting upload Dec 28 05:37:26.528457761 (542h8m26.399540806s) [upload=475418]: Uploading segments... Dec 28 05:37:26.528522084 (542h8m26.399605129s) [upload=475418]: Got next segment. Inline: false Dec 28 05:37:26.528557982 (542h8m26.399641027s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:37:26.528563753 (542h8m26.399646798s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:37:26.774247715 (542h8m26.64533076s) [seg_pos={0 0}, upload=475418]: Begin upload segment... Dec 28 05:37:26.774303080 (542h8m26.645386115s) [seg_pos={0 0}, upload=475418]: Done begin upload segment. Dec 28 05:37:26.774335341 (542h8m26.645418376s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.774436564 (542h8m26.645519609s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775076573 (542h8m26.646159608s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775533002 (542h8m26.646616047s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775883490 (542h8m26.646966525s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776314460 (542h8m26.647397495s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776642525 (542h8m26.64772556s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776974116 (542h8m26.648057161s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778179121 (542h8m26.649262166s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778471938 (542h8m26.649554983s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.779537498 (542h8m26.650620543s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.780745327 (542h8m26.651828372s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.782469380 (542h8m26.653552415s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.783773834 (542h8m26.654856869s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.784949913 (542h8m26.656032948s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.786010794 (542h8m26.657093839s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:28.505308401 (542h8m28.376391436s) [upload=475418]: Next returned nil segment Dec 28 05:37:28.505322969 (542h8m28.376406004s) [upload=475418]: Waiting for error group managing segments... Dec 28 05:37:28.795090121 (542h8m28.666173157s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797384000 (542h8m28.668467035s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797973863 (542h8m28.669056908s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.806260141 (542h8m28.677343176s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.808582935 (542h8m28.67966597s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811020656 (542h8m28.682103691s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811576304 (542h8m28.682659349s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.813730496 (542h8m28.684813542s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.026606336 (542h8m28.897689371s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.072280195 (542h8m28.943363241s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.319295923 (542h8m29.190378958s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.703500991 (542h8m29.574584026s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736686772 (542h8m29.607769817s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736711710 (542h8m29.607794745s) [seg_pos={0 0}, upload=475418]: Segment reached optimal threshold of 13 pieces. Dec 28 05:37:29.736898094 (542h8m29.607981129s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.736904747 (542h8m29.607987782s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737028763 (542h8m29.608111809s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737034053 (542h8m29.608117088s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737172086 (542h8m29.608255121s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737177727 (542h8m29.608260772s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737204217 (542h8m29.608287253s) [seg_pos={0 0}, upload=475418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:37:29.737271245 (542h8m29.60835428s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:37:29.737275834 (542h8m29.608358869s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:37:30.033350437 (542h8m29.904433482s) [upload=475418]: Done uploading segments... Dec 28 05:37:30.033373751 (542h8m29.904456786s) [upload=475418]: Upload finished. err: <nil> Duration: 5.771090111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X] duration: 5.001708772s trace: https://jaeger.dev.storj.io/trace/768e65a7989ace8 FAIL (2025-12-28 05:38:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001734189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............................X.........X...X.......] duration: 35.993075902s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 35.993091692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X.] duration: 6.146130794s trace: https://jaeger.dev.storj.io/trace/505bab7549ef4ee6 FAIL (2025-12-28 05:38:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A) context deadline exceeded ======LOGS====== Dec 28 05:37:26.527890652 (542h8m26.398973707s) [upload=475418]: Starting upload Dec 28 05:37:26.527974341 (542h8m26.399057386s) [upload=475418]: Done starting upload Dec 28 05:37:26.528457761 (542h8m26.399540806s) [upload=475418]: Uploading segments... Dec 28 05:37:26.528522084 (542h8m26.399605129s) [upload=475418]: Got next segment. Inline: false Dec 28 05:37:26.528557982 (542h8m26.399641027s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:37:26.528563753 (542h8m26.399646798s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:37:26.774247715 (542h8m26.64533076s) [seg_pos={0 0}, upload=475418]: Begin upload segment... Dec 28 05:37:26.774303080 (542h8m26.645386115s) [seg_pos={0 0}, upload=475418]: Done begin upload segment. Dec 28 05:37:26.774335341 (542h8m26.645418376s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.774436564 (542h8m26.645519609s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775076573 (542h8m26.646159608s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775533002 (542h8m26.646616047s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775883490 (542h8m26.646966525s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776314460 (542h8m26.647397495s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776642525 (542h8m26.64772556s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776974116 (542h8m26.648057161s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778179121 (542h8m26.649262166s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778471938 (542h8m26.649554983s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.779537498 (542h8m26.650620543s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.780745327 (542h8m26.651828372s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.782469380 (542h8m26.653552415s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.783773834 (542h8m26.654856869s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.784949913 (542h8m26.656032948s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.786010794 (542h8m26.657093839s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:28.505308401 (542h8m28.376391436s) [upload=475418]: Next returned nil segment Dec 28 05:37:28.505322969 (542h8m28.376406004s) [upload=475418]: Waiting for error group managing segments... Dec 28 05:37:28.795090121 (542h8m28.666173157s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797384000 (542h8m28.668467035s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797973863 (542h8m28.669056908s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.806260141 (542h8m28.677343176s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.808582935 (542h8m28.67966597s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811020656 (542h8m28.682103691s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811576304 (542h8m28.682659349s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.813730496 (542h8m28.684813542s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.026606336 (542h8m28.897689371s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.072280195 (542h8m28.943363241s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.319295923 (542h8m29.190378958s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.703500991 (542h8m29.574584026s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736686772 (542h8m29.607769817s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736711710 (542h8m29.607794745s) [seg_pos={0 0}, upload=475418]: Segment reached optimal threshold of 13 pieces. Dec 28 05:37:29.736898094 (542h8m29.607981129s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.736904747 (542h8m29.607987782s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737028763 (542h8m29.608111809s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737034053 (542h8m29.608117088s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737172086 (542h8m29.608255121s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737177727 (542h8m29.608260772s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737204217 (542h8m29.608287253s) [seg_pos={0 0}, upload=475418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:37:29.737271245 (542h8m29.60835428s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:37:29.737275834 (542h8m29.608358869s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:37:30.033350437 (542h8m29.904433482s) [upload=475418]: Done uploading segments... Dec 28 05:37:30.033373751 (542h8m29.904456786s) [upload=475418]: Upload finished. err: <nil> Duration: 6.146161151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X] duration: 30.002927228s trace: https://jaeger.dev.storj.io/trace/505bab7549ef4ee6 FAIL (2025-12-28 05:37:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A) context deadline exceeded ======LOGS====== Dec 28 05:37:26.527890652 (542h8m26.398973707s) [upload=475418]: Starting upload Dec 28 05:37:26.527974341 (542h8m26.399057386s) [upload=475418]: Done starting upload Dec 28 05:37:26.528457761 (542h8m26.399540806s) [upload=475418]: Uploading segments... Dec 28 05:37:26.528522084 (542h8m26.399605129s) [upload=475418]: Got next segment. Inline: false Dec 28 05:37:26.528557982 (542h8m26.399641027s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:37:26.528563753 (542h8m26.399646798s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:37:26.774247715 (542h8m26.64533076s) [seg_pos={0 0}, upload=475418]: Begin upload segment... Dec 28 05:37:26.774303080 (542h8m26.645386115s) [seg_pos={0 0}, upload=475418]: Done begin upload segment. Dec 28 05:37:26.774335341 (542h8m26.645418376s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.774436564 (542h8m26.645519609s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775076573 (542h8m26.646159608s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775533002 (542h8m26.646616047s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.775883490 (542h8m26.646966525s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776314460 (542h8m26.647397495s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776642525 (542h8m26.64772556s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.776974116 (542h8m26.648057161s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778179121 (542h8m26.649262166s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.778471938 (542h8m26.649554983s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.779537498 (542h8m26.650620543s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.780745327 (542h8m26.651828372s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.782469380 (542h8m26.653552415s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.783773834 (542h8m26.654856869s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.784949913 (542h8m26.656032948s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:26.786010794 (542h8m26.657093839s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Uploading piece... Dec 28 05:37:28.505308401 (542h8m28.376391436s) [upload=475418]: Next returned nil segment Dec 28 05:37:28.505322969 (542h8m28.376406004s) [upload=475418]: Waiting for error group managing segments... Dec 28 05:37:28.795090121 (542h8m28.666173157s) [noise=true , address=145.79.227.27:4691 , piece_id=C7ZCJIAPSCHUC6YATL5SFTZNLNL3Z6FKWFMCDOKXD62KUWGYITEA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797384000 (542h8m28.668467035s) [noise=true , address=145.79.227.11:4732 , piece_id=QIBGECDP27VEXD3CNBX23W3UMVXCBOEXH5AV3ZEANGBMBRE76LRA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.797973863 (542h8m28.669056908s) [noise=true , address=145.79.227.39:4015 , piece_id=7Z5N4LS22HCGSDPDWASFBXLJ27CK3V7H4URKOO6GFHPFKMBVUH3Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.806260141 (542h8m28.677343176s) [noise=true , address=145.79.227.4:4186 , piece_id=NHD3CD263VH7XAYUVLXG4TTCSDHZHCMXDRYNT6W7FN3PH5J77KQQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.808582935 (542h8m28.67966597s) [noise=true , address=145.79.227.45:4229 , piece_id=KSLIOS6PWDKHFG7BWKW3N5CKOFVYIVPA5FTOF65PGWZBQ4KYSFFA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811020656 (542h8m28.682103691s) [noise=true , address=145.79.227.25:4617 , piece_id=IAXEJC65MEH54BISTKA6GXNPGUXYL2P5US3IS2V5A2Y3IBX3GQIQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.811576304 (542h8m28.682659349s) [noise=true , address=145.79.227.2:4115 , piece_id=KYQPSTOFWN65A3V6SVO5RWYKDCBLLMO7TFBXEW3XZXENPFBPO4YA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:28.813730496 (542h8m28.684813542s) [noise=true , address=145.79.227.38:4977 , piece_id=UKTZNYV7VLLDTF2GAQLUPBWOWYJN6IGSIXFWGU46G4UKLJ5JWRBA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.026606336 (542h8m28.897689371s) [noise=true , address=145.79.227.49:4337 , piece_id=2XZTQ3PXOKNOMU7BJTOUXQZCDXJWNDX3G7PH52RXTEK5AOHHYULQ, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.072280195 (542h8m28.943363241s) [noise=true , address=45.91.116.5:4297 , piece_id=2HG7QJ6D5XC3PZ2RFXA5LS3PRX4TUE7MCFA4UJK2QD5GUDCT7M6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.319295923 (542h8m29.190378958s) [noise=true , address=145.79.227.34:4905 , piece_id=ROMY6GQGUYK3UAKWIBDOU5K5I6SU4AHJUKXFE7UWS3H34YJG4G6A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.703500991 (542h8m29.574584026s) [noise=true , address=77.90.31.10:4519 , piece_id=RVGQRZXHAHY4YG5T24FDMIHAHMVKE347G2FPEUVALXELOG4DMX5A, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736686772 (542h8m29.607769817s) [noise=true , address=145.79.227.35:4483 , piece_id=BWLLDG34Q5UMQXE4NXKS7UTSFLHMOO52EDF6RF3EWGMPPTYRAVCA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: <nil> Dec 28 05:37:29.736711710 (542h8m29.607794745s) [seg_pos={0 0}, upload=475418]: Segment reached optimal threshold of 13 pieces. Dec 28 05:37:29.736898094 (542h8m29.607981129s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.736904747 (542h8m29.607987782s) [noise=true , address=145.79.227.44:4193 , piece_id=62I2I5UQXZA3LR6JZTGNNMTIRY64SJ3IVRTACJCO3OLYPYSIQQ6Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737028763 (542h8m29.608111809s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737034053 (542h8m29.608117088s) [noise=true , address=45.91.116.7:4011 , piece_id=AVNSJCEMD54NEX6JW4VUUJP7YPFWFHC5FXN3M5EIHKFXO67AUP5Q, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737172086 (542h8m29.608255121s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 05:37:29.737177727 (542h8m29.608260772s) [noise=true , address=77.90.31.17:4770 , piece_id=N3H63RNJIOSMA3ZBD2WTQVNWQV4NCKQZR376NYQKJEEPAIXHZRYA, seg_pos={0 0}, upload=475418]: Long tail cancellation due to optimal threshold Dec 28 05:37:29.737204217 (542h8m29.608287253s) [seg_pos={0 0}, upload=475418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:37:29.737271245 (542h8m29.60835428s) [upload=475418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:37:29.737275834 (542h8m29.608358869s) [upload=475418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:37:30.033350437 (542h8m29.904433482s) [upload=475418]: Done uploading segments... Dec 28 05:37:30.033373751 (542h8m29.904456786s) [upload=475418]: Upload finished. err: <nil> Duration: 30.002968546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.............................X.........X...X......] duration: 38.119945818s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 38.11997856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.45247693s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:37:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.452698031s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X....] duration: 5.001438699s trace: https://jaeger.dev.storj.io/trace/2c215b2096ffae82 FAIL (2025-12-28 05:37:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001482822s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X...] duration: 5.0017929s trace: https://jaeger.dev.storj.io/trace/2c215b2096ffae82 FAIL (2025-12-28 05:36:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001820442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.............................X.........X...X.....] duration: 47.783471715s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 47.78349003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.25604956s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:36:32 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.25607667s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.] duration: 6.074971243s trace: https://jaeger.dev.storj.io/trace/e8ea8c393cd5c57 FAIL (2025-12-28 05:36:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A) context deadline exceeded ======LOGS====== 5:22.774906849 (542h6m22.645989905s) [upload=475388]: Starting upload Dec 28 05:35:22.774983806 (542h6m22.646066871s) [upload=475388]: Done starting upload Dec 28 05:35:22.775173718 (542h6m22.646256763s) [upload=475388]: Uploading segments... Dec 28 05:35:22.775289178 (542h6m22.646372223s) [upload=475388]: Got next segment. Inline: false Dec 28 05:35:22.775360644 (542h6m22.646443679s) [upload=475388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:35:22.775366875 (542h6m22.64644991s) [upload=475388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:35:23.021388189 (542h6m22.892471224s) [seg_pos={0 0}, upload=475388]: Begin upload segment... Dec 28 05:35:23.021439277 (542h6m22.892522312s) [seg_pos={0 0}, upload=475388]: Done begin upload segment. Dec 28 05:35:23.021463243 (542h6m22.892546288s) [noise=true , address=145.79.227.31:4799 , piece_id=BPOLBCOP3YBPV5HYB5SRTKGUVBMBDNDPPTGIVHX6M7TLRLB7RJAA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.022430414 (542h6m22.893513459s) [noise=true , address=145.79.227.5:4221 , piece_id=6RYNO5DE4NWCKK2IHO3JXCCL65PLS2F4PV2FW2STBQS2CKVASBAQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023022731 (542h6m22.894105766s) [noise=true , address=145.79.227.18:4007 , piece_id=DJP4QKSVEQCY77CA333HOVO4MSE65KFW64SDTM3QNKT4AAZHYA2A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023475083 (542h6m22.894558128s) [noise=true , address=145.79.227.14:4837 , piece_id=QBAZSOVLIAREPTBPNF4CG2EA6RI6RKH5XQJMQ4WP7EECFEYBBOTA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023858373 (542h6m22.894941418s) [noise=true , address=145.79.227.15:4873 , piece_id=Y3UBSBRIJFVZIT2T3W2UFQ3NU272MWZAFJLZQ4WS36ZSS6SFBR7A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.024273122 (542h6m22.895356168s) [noise=true , address=45.91.116.7:4010 , piece_id=BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.024771803 (542h6m22.895854838s) [noise=true , address=145.79.227.42:4123 , piece_id=EUQTUORYZXG7A6ADVIOENSRYCLJK7PIMKDNFHHSETMVSSCMIX66A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.025250203 (542h6m22.896333248s) [noise=true , address=145.79.227.45:4231 , piece_id=GU7XWQSZYWBPRRJ5TDCXXR3UCGMHJN2QV44IXOXCZLYEJX4PBW4A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.027368837 (542h6m22.898451872s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.029972385 (542h6m22.90105542s) [noise=true , address=145.79.227.36:4517 , piece_id=SHVEVNATKYLTH6GBIUV6EXURAIJK6M74JWWSSPYAV2SPKUIER7TQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.032143559 (542h6m22.903226604s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.034079826 (542h6m22.905162871s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.036034558 (542h6m22.907117603s) [noise=true , address=45.91.116.9:4261 , piece_id=HWRNNWIYJCSGJV7SBSFVU6MMCV6YJIPWCOE2ZEMCFWR3N6S6X76A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.037888198 (542h6m22.908971233s) [noise=true , address=77.90.31.14:4663 , piece_id=6ID375FKA6VLOFH554UWNLDDMDEJZJU7NFQW7DKPSFLJ7UAZZBXQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.037910220 (542h6m22.908993265s) [noise=true , address=45.91.116.12:4226 , piece_id=JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.040737343 (542h6m22.911820378s) [noise=true , address=145.79.227.50:4375 , piece_id=FKME2JVMVC27IUUSHSLYFD54YF6IX2LUQRJ4RHZNHRBWTXSAYUTQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:24.729418693 (542h6m24.600501748s) [upload=475388]: Next returned nil segment Dec 28 05:35:24.729436948 (542h6m24.600519993s) [upload=475388]: Waiting for error group managing segments... Dec 28 05:35:25.031348626 (542h6m24.902431661s) [noise=true , address=145.79.227.45:4231 , piece_id=GU7XWQSZYWBPRRJ5TDCXXR3UCGMHJN2QV44IXOXCZLYEJX4PBW4A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.041878788 (542h6m24.912961833s) [noise=true , address=145.79.227.36:4517 , piece_id=SHVEVNATKYLTH6GBIUV6EXURAIJK6M74JWWSSPYAV2SPKUIER7TQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.047764585 (542h6m24.91884763s) [noise=true , address=145.79.227.5:4221 , piece_id=6RYNO5DE4NWCKK2IHO3JXCCL65PLS2F4PV2FW2STBQS2CKVASBAQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.057117704 (542h6m24.92820075s) [noise=true , address=145.79.227.18:4007 , piece_id=DJP4QKSVEQCY77CA333HOVO4MSE65KFW64SDTM3QNKT4AAZHYA2A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.058149278 (542h6m24.929232313s) [noise=true , address=145.79.227.50:4375 , piece_id=FKME2JVMVC27IUUSHSLYFD54YF6IX2LUQRJ4RHZNHRBWTXSAYUTQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.058941707 (542h6m24.930024752s) [noise=true , address=145.79.227.31:4799 , piece_id=BPOLBCOP3YBPV5HYB5SRTKGUVBMBDNDPPTGIVHX6M7TLRLB7RJAA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.059851449 (542h6m24.930934495s) [noise=true , address=145.79.227.42:4123 , piece_id=EUQTUORYZXG7A6ADVIOENSRYCLJK7PIMKDNFHHSETMVSSCMIX66A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.070711349 (542h6m24.941794384s) [noise=true , address=145.79.227.15:4873 , piece_id=Y3UBSBRIJFVZIT2T3W2UFQ3NU272MWZAFJLZQ4WS36ZSS6SFBR7A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.171977305 (542h6m25.043060351s) [noise=true , address=145.79.227.14:4837 , piece_id=QBAZSOVLIAREPTBPNF4CG2EA6RI6RKH5XQJMQ4WP7EECFEYBBOTA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.234327872 (542h6m25.105410917s) [noise=true , address=77.90.31.14:4663 , piece_id=6ID375FKA6VLOFH554UWNLDDMDEJZJU7NFQW7DKPSFLJ7UAZZBXQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.323024244 (542h6m25.194107289s) [noise=true , address=45.91.116.12:4226 , piece_id=JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.326366768 (542h6m25.197449813s) [noise=true , address=45.91.116.7:4010 , piece_id=BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.426946557 (542h6m25.298029603s) [noise=true , address=45.91.116.9:4261 , piece_id=HWRNNWIYJCSGJV7SBSFVU6MMCV6YJIPWCOE2ZEMCFWR3N6S6X76A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.426969842 (542h6m25.298052887s) [seg_pos={0 0}, upload=475388]: Segment reached optimal threshold of 13 pieces. Dec 28 05:35:25.427190582 (542h6m25.298273627s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection Dec 28 05:35:25.427201623 (542h6m25.298284678s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427265675 (542h6m25.29834872s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 05:35:25.427278530 (542h6m25.298361565s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427572449 (542h6m25.298655494s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:52074->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 05:35:25.427582969 (542h6m25.298666014s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427603759 (542h6m25.298686804s) [seg_pos={0 0}, upload=475388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:35:25.427773192 (542h6m25.298856237s) [upload=475388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:35:25.427781567 (542h6m25.298864613s) [upload=475388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:35:25.716674356 (542h6m25.587757391s) [upload=475388]: Done uploading segments... Dec 28 05:35:25.716696589 (542h6m25.587779664s) [upload=475388]: Upload finished. err: <nil> Duration: 6.074998805s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.254475224s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:35:55 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.254502655s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.............................X.........X...X....] duration: 36.634008657s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 36.634026641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X] duration: 30.00308233s trace: https://jaeger.dev.storj.io/trace/e8ea8c393cd5c57 FAIL (2025-12-28 05:35:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A) context deadline exceeded ======LOGS====== 5:22.774906849 (542h6m22.645989905s) [upload=475388]: Starting upload Dec 28 05:35:22.774983806 (542h6m22.646066871s) [upload=475388]: Done starting upload Dec 28 05:35:22.775173718 (542h6m22.646256763s) [upload=475388]: Uploading segments... Dec 28 05:35:22.775289178 (542h6m22.646372223s) [upload=475388]: Got next segment. Inline: false Dec 28 05:35:22.775360644 (542h6m22.646443679s) [upload=475388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:35:22.775366875 (542h6m22.64644991s) [upload=475388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:35:23.021388189 (542h6m22.892471224s) [seg_pos={0 0}, upload=475388]: Begin upload segment... Dec 28 05:35:23.021439277 (542h6m22.892522312s) [seg_pos={0 0}, upload=475388]: Done begin upload segment. Dec 28 05:35:23.021463243 (542h6m22.892546288s) [noise=true , address=145.79.227.31:4799 , piece_id=BPOLBCOP3YBPV5HYB5SRTKGUVBMBDNDPPTGIVHX6M7TLRLB7RJAA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.022430414 (542h6m22.893513459s) [noise=true , address=145.79.227.5:4221 , piece_id=6RYNO5DE4NWCKK2IHO3JXCCL65PLS2F4PV2FW2STBQS2CKVASBAQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023022731 (542h6m22.894105766s) [noise=true , address=145.79.227.18:4007 , piece_id=DJP4QKSVEQCY77CA333HOVO4MSE65KFW64SDTM3QNKT4AAZHYA2A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023475083 (542h6m22.894558128s) [noise=true , address=145.79.227.14:4837 , piece_id=QBAZSOVLIAREPTBPNF4CG2EA6RI6RKH5XQJMQ4WP7EECFEYBBOTA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.023858373 (542h6m22.894941418s) [noise=true , address=145.79.227.15:4873 , piece_id=Y3UBSBRIJFVZIT2T3W2UFQ3NU272MWZAFJLZQ4WS36ZSS6SFBR7A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.024273122 (542h6m22.895356168s) [noise=true , address=45.91.116.7:4010 , piece_id=BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.024771803 (542h6m22.895854838s) [noise=true , address=145.79.227.42:4123 , piece_id=EUQTUORYZXG7A6ADVIOENSRYCLJK7PIMKDNFHHSETMVSSCMIX66A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.025250203 (542h6m22.896333248s) [noise=true , address=145.79.227.45:4231 , piece_id=GU7XWQSZYWBPRRJ5TDCXXR3UCGMHJN2QV44IXOXCZLYEJX4PBW4A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.027368837 (542h6m22.898451872s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.029972385 (542h6m22.90105542s) [noise=true , address=145.79.227.36:4517 , piece_id=SHVEVNATKYLTH6GBIUV6EXURAIJK6M74JWWSSPYAV2SPKUIER7TQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.032143559 (542h6m22.903226604s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.034079826 (542h6m22.905162871s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.036034558 (542h6m22.907117603s) [noise=true , address=45.91.116.9:4261 , piece_id=HWRNNWIYJCSGJV7SBSFVU6MMCV6YJIPWCOE2ZEMCFWR3N6S6X76A, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.037888198 (542h6m22.908971233s) [noise=true , address=77.90.31.14:4663 , piece_id=6ID375FKA6VLOFH554UWNLDDMDEJZJU7NFQW7DKPSFLJ7UAZZBXQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.037910220 (542h6m22.908993265s) [noise=true , address=45.91.116.12:4226 , piece_id=JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:23.040737343 (542h6m22.911820378s) [noise=true , address=145.79.227.50:4375 , piece_id=FKME2JVMVC27IUUSHSLYFD54YF6IX2LUQRJ4RHZNHRBWTXSAYUTQ, seg_pos={0 0}, upload=475388]: Uploading piece... Dec 28 05:35:24.729418693 (542h6m24.600501748s) [upload=475388]: Next returned nil segment Dec 28 05:35:24.729436948 (542h6m24.600519993s) [upload=475388]: Waiting for error group managing segments... Dec 28 05:35:25.031348626 (542h6m24.902431661s) [noise=true , address=145.79.227.45:4231 , piece_id=GU7XWQSZYWBPRRJ5TDCXXR3UCGMHJN2QV44IXOXCZLYEJX4PBW4A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.041878788 (542h6m24.912961833s) [noise=true , address=145.79.227.36:4517 , piece_id=SHVEVNATKYLTH6GBIUV6EXURAIJK6M74JWWSSPYAV2SPKUIER7TQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.047764585 (542h6m24.91884763s) [noise=true , address=145.79.227.5:4221 , piece_id=6RYNO5DE4NWCKK2IHO3JXCCL65PLS2F4PV2FW2STBQS2CKVASBAQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.057117704 (542h6m24.92820075s) [noise=true , address=145.79.227.18:4007 , piece_id=DJP4QKSVEQCY77CA333HOVO4MSE65KFW64SDTM3QNKT4AAZHYA2A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.058149278 (542h6m24.929232313s) [noise=true , address=145.79.227.50:4375 , piece_id=FKME2JVMVC27IUUSHSLYFD54YF6IX2LUQRJ4RHZNHRBWTXSAYUTQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.058941707 (542h6m24.930024752s) [noise=true , address=145.79.227.31:4799 , piece_id=BPOLBCOP3YBPV5HYB5SRTKGUVBMBDNDPPTGIVHX6M7TLRLB7RJAA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.059851449 (542h6m24.930934495s) [noise=true , address=145.79.227.42:4123 , piece_id=EUQTUORYZXG7A6ADVIOENSRYCLJK7PIMKDNFHHSETMVSSCMIX66A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.070711349 (542h6m24.941794384s) [noise=true , address=145.79.227.15:4873 , piece_id=Y3UBSBRIJFVZIT2T3W2UFQ3NU272MWZAFJLZQ4WS36ZSS6SFBR7A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.171977305 (542h6m25.043060351s) [noise=true , address=145.79.227.14:4837 , piece_id=QBAZSOVLIAREPTBPNF4CG2EA6RI6RKH5XQJMQ4WP7EECFEYBBOTA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.234327872 (542h6m25.105410917s) [noise=true , address=77.90.31.14:4663 , piece_id=6ID375FKA6VLOFH554UWNLDDMDEJZJU7NFQW7DKPSFLJ7UAZZBXQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.323024244 (542h6m25.194107289s) [noise=true , address=45.91.116.12:4226 , piece_id=JHWUJOO32WY2XFF43C4DXE77Y7KW47ZZ6A7Q4MIK2KPSFU6HCH7Q, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.326366768 (542h6m25.197449813s) [noise=true , address=45.91.116.7:4010 , piece_id=BKEU4SMRZ5P7SXFGINQAI2X2SNIZNFDEU4KNFPB4UL2NX54LJP7A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.426946557 (542h6m25.298029603s) [noise=true , address=45.91.116.9:4261 , piece_id=HWRNNWIYJCSGJV7SBSFVU6MMCV6YJIPWCOE2ZEMCFWR3N6S6X76A, seg_pos={0 0}, upload=475388]: Done uploading piece. err: <nil> Dec 28 05:35:25.426969842 (542h6m25.298052887s) [seg_pos={0 0}, upload=475388]: Segment reached optimal threshold of 13 pieces. Dec 28 05:35:25.427190582 (542h6m25.298273627s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51844->45.91.116.4:4940: use of closed network connection Dec 28 05:35:25.427201623 (542h6m25.298284678s) [noise=true , address=45.91.116.4:4940 , piece_id=LG6WMDGIPUEBH5P753GYXOE7KVY4OCWZT2ICGGHBUARYBKNWLQCA, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427265675 (542h6m25.29834872s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 05:35:25.427278530 (542h6m25.298361565s) [noise=true , address=145.79.227.43:4159 , piece_id=CEE2ZUHWIHG5SHK7UMVNNE5KLVXXBC7VX7PCNUUTCG3Z3BJZF3TQ, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427572449 (542h6m25.298655494s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:52074->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 05:35:25.427582969 (542h6m25.298666014s) [noise=true , address=45.91.116.8:4047 , piece_id=TTUE7VSNNAK5LQPOLSVQSS2FFWOOA3PVI3LWE2R3GEAZKLWHI6CA, seg_pos={0 0}, upload=475388]: Long tail cancellation due to optimal threshold Dec 28 05:35:25.427603759 (542h6m25.298686804s) [seg_pos={0 0}, upload=475388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:35:25.427773192 (542h6m25.298856237s) [upload=475388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:35:25.427781567 (542h6m25.298864613s) [upload=475388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:35:25.716674356 (542h6m25.587757391s) [upload=475388]: Done uploading segments... Dec 28 05:35:25.716696589 (542h6m25.587779664s) [upload=475388]: Upload finished. err: <nil> Duration: 30.00310349s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X..] duration: 5.001901002s trace: https://jaeger.dev.storj.io/trace/2c215b2096ffae82 FAIL (2025-12-28 05:35:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001928193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.............................X.........X...X...] duration: 36.444038567s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:35:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 36.444068393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X.] duration: 5.006205457s trace: https://jaeger.dev.storj.io/trace/2c215b2096ffae82 FAIL (2025-12-28 05:35:04 UTC) orbiter warning: context deadline exceeded Duration: 5.006226547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.03498011s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:35:00 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.035013594s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X...] duration: 5.958881584s trace: https://jaeger.dev.storj.io/trace/4c1d92962d1bb6f7 FAIL (2025-12-28 05:34:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:32:04.141520669 (542h3m4.012603715s) [upload=475340]: Starting upload Dec 28 05:32:04.141623095 (542h3m4.01270614s) [upload=475340]: Done starting upload Dec 28 05:32:04.141722734 (542h3m4.01280578s) [upload=475340]: Uploading segments... Dec 28 05:32:04.141929738 (542h3m4.013012783s) [upload=475340]: Got next segment. Inline: false Dec 28 05:32:04.141974574 (542h3m4.013057619s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:32:04.141982298 (542h3m4.013065343s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:32:04.388024523 (542h3m4.259107578s) [seg_pos={0 0}, upload=475340]: Begin upload segment... Dec 28 05:32:04.388096250 (542h3m4.259179295s) [seg_pos={0 0}, upload=475340]: Done begin upload segment. Dec 28 05:32:04.388109795 (542h3m4.25919284s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388140824 (542h3m4.259223869s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388700881 (542h3m4.259783936s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389402006 (542h3m4.260485051s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389712988 (542h3m4.260796033s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389857513 (542h3m4.260940558s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390184426 (542h3m4.261267471s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390278695 (542h3m4.26136173s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390689687 (542h3m4.261772733s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390799647 (542h3m4.261882692s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391131820 (542h3m4.262214865s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391155024 (542h3m4.262238069s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391510411 (542h3m4.262593446s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391985545 (542h3m4.26306859s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392440200 (542h3m4.263523246s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392875239 (542h3m4.263958284s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:06.306196329 (542h3m6.177279364s) [upload=475340]: Next returned nil segment Dec 28 05:32:06.306210625 (542h3m6.17729367s) [upload=475340]: Waiting for error group managing segments... Dec 28 05:32:06.663243664 (542h3m6.534326709s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.701106650 (542h3m6.572189686s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.729004512 (542h3m6.600087558s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.795888771 (542h3m6.666971816s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.925031540 (542h3m6.796114575s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.137489875 (542h3m7.00857292s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.247045706 (542h3m7.118128751s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.421350219 (542h3m7.292433264s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.474374597 (542h3m7.345457632s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.046066699 (542h3m7.917149744s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.381184037 (542h3m8.252267092s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.503852607 (542h3m8.374935652s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053690045 (542h3m8.92477308s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053709042 (542h3m8.924792077s) [seg_pos={0 0}, upload=475340]: Segment reached optimal threshold of 13 pieces. Dec 28 05:32:09.054106398 (542h3m8.925189443s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: rpc: dial tcp 145.79.227.5:4222: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054117039 (542h3m8.925200074s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054285930 (542h3m8.925368965s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: read tcp 188.166.225.170:37674->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054293886 (542h3m8.925376931s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054329273 (542h3m8.925412308s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:46006->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054343089 (542h3m8.925426134s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054372255 (542h3m8.92545529s) [seg_pos={0 0}, upload=475340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:32:09.054466905 (542h3m8.92554995s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:09.054473147 (542h3m8.925556182s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:09.348157750 (542h3m9.219240805s) [upload=475340]: Done uploading segments... Dec 28 05:32:09.348180353 (542h3m9.219263388s) [upload=475340]: Upload finished. err: <nil> Duration: 5.958909997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............................X.........X...X..] duration: 38.333968756s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 38.333998181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...X] duration: 5.001510908s trace: https://jaeger.dev.storj.io/trace/2c215b2096ffae82 FAIL (2025-12-28 05:34:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001547829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.182054981s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:34:12 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.182080588s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X..] duration: 5.953372827s trace: https://jaeger.dev.storj.io/trace/4c1d92962d1bb6f7 FAIL (2025-12-28 05:33:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:32:04.141520669 (542h3m4.012603715s) [upload=475340]: Starting upload Dec 28 05:32:04.141623095 (542h3m4.01270614s) [upload=475340]: Done starting upload Dec 28 05:32:04.141722734 (542h3m4.01280578s) [upload=475340]: Uploading segments... Dec 28 05:32:04.141929738 (542h3m4.013012783s) [upload=475340]: Got next segment. Inline: false Dec 28 05:32:04.141974574 (542h3m4.013057619s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:32:04.141982298 (542h3m4.013065343s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:32:04.388024523 (542h3m4.259107578s) [seg_pos={0 0}, upload=475340]: Begin upload segment... Dec 28 05:32:04.388096250 (542h3m4.259179295s) [seg_pos={0 0}, upload=475340]: Done begin upload segment. Dec 28 05:32:04.388109795 (542h3m4.25919284s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388140824 (542h3m4.259223869s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388700881 (542h3m4.259783936s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389402006 (542h3m4.260485051s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389712988 (542h3m4.260796033s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389857513 (542h3m4.260940558s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390184426 (542h3m4.261267471s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390278695 (542h3m4.26136173s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390689687 (542h3m4.261772733s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390799647 (542h3m4.261882692s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391131820 (542h3m4.262214865s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391155024 (542h3m4.262238069s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391510411 (542h3m4.262593446s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391985545 (542h3m4.26306859s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392440200 (542h3m4.263523246s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392875239 (542h3m4.263958284s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:06.306196329 (542h3m6.177279364s) [upload=475340]: Next returned nil segment Dec 28 05:32:06.306210625 (542h3m6.17729367s) [upload=475340]: Waiting for error group managing segments... Dec 28 05:32:06.663243664 (542h3m6.534326709s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.701106650 (542h3m6.572189686s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.729004512 (542h3m6.600087558s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.795888771 (542h3m6.666971816s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.925031540 (542h3m6.796114575s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.137489875 (542h3m7.00857292s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.247045706 (542h3m7.118128751s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.421350219 (542h3m7.292433264s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.474374597 (542h3m7.345457632s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.046066699 (542h3m7.917149744s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.381184037 (542h3m8.252267092s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.503852607 (542h3m8.374935652s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053690045 (542h3m8.92477308s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053709042 (542h3m8.924792077s) [seg_pos={0 0}, upload=475340]: Segment reached optimal threshold of 13 pieces. Dec 28 05:32:09.054106398 (542h3m8.925189443s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: rpc: dial tcp 145.79.227.5:4222: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054117039 (542h3m8.925200074s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054285930 (542h3m8.925368965s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: read tcp 188.166.225.170:37674->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054293886 (542h3m8.925376931s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054329273 (542h3m8.925412308s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:46006->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054343089 (542h3m8.925426134s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054372255 (542h3m8.92545529s) [seg_pos={0 0}, upload=475340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:32:09.054466905 (542h3m8.92554995s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:09.054473147 (542h3m8.925556182s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:09.348157750 (542h3m9.219240805s) [upload=475340]: Done uploading segments... Dec 28 05:32:09.348180353 (542h3m9.219263388s) [upload=475340]: Upload finished. err: <nil> Duration: 5.95339584s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.955534462s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:33:41 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 4.955559941s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.............................X.........X...X.] duration: 43.100123944s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:33:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 43.100149361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.254439458s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:33:06 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.254480296s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X...] duration: 3.702345701s trace: https://jaeger.dev.storj.io/trace/74b7dbd71910e0e2 FAIL (2025-12-28 05:33:05 UTC) orbiter warning: context deadline exceeded Duration: 3.702412127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X.] duration: 6.20692989s trace: https://jaeger.dev.storj.io/trace/4c1d92962d1bb6f7 FAIL (2025-12-28 05:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:32:04.141520669 (542h3m4.012603715s) [upload=475340]: Starting upload Dec 28 05:32:04.141623095 (542h3m4.01270614s) [upload=475340]: Done starting upload Dec 28 05:32:04.141722734 (542h3m4.01280578s) [upload=475340]: Uploading segments... Dec 28 05:32:04.141929738 (542h3m4.013012783s) [upload=475340]: Got next segment. Inline: false Dec 28 05:32:04.141974574 (542h3m4.013057619s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:32:04.141982298 (542h3m4.013065343s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:32:04.388024523 (542h3m4.259107578s) [seg_pos={0 0}, upload=475340]: Begin upload segment... Dec 28 05:32:04.388096250 (542h3m4.259179295s) [seg_pos={0 0}, upload=475340]: Done begin upload segment. Dec 28 05:32:04.388109795 (542h3m4.25919284s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388140824 (542h3m4.259223869s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388700881 (542h3m4.259783936s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389402006 (542h3m4.260485051s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389712988 (542h3m4.260796033s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389857513 (542h3m4.260940558s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390184426 (542h3m4.261267471s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390278695 (542h3m4.26136173s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390689687 (542h3m4.261772733s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390799647 (542h3m4.261882692s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391131820 (542h3m4.262214865s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391155024 (542h3m4.262238069s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391510411 (542h3m4.262593446s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391985545 (542h3m4.26306859s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392440200 (542h3m4.263523246s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392875239 (542h3m4.263958284s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:06.306196329 (542h3m6.177279364s) [upload=475340]: Next returned nil segment Dec 28 05:32:06.306210625 (542h3m6.17729367s) [upload=475340]: Waiting for error group managing segments... Dec 28 05:32:06.663243664 (542h3m6.534326709s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.701106650 (542h3m6.572189686s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.729004512 (542h3m6.600087558s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.795888771 (542h3m6.666971816s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.925031540 (542h3m6.796114575s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.137489875 (542h3m7.00857292s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.247045706 (542h3m7.118128751s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.421350219 (542h3m7.292433264s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.474374597 (542h3m7.345457632s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.046066699 (542h3m7.917149744s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.381184037 (542h3m8.252267092s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.503852607 (542h3m8.374935652s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053690045 (542h3m8.92477308s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053709042 (542h3m8.924792077s) [seg_pos={0 0}, upload=475340]: Segment reached optimal threshold of 13 pieces. Dec 28 05:32:09.054106398 (542h3m8.925189443s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: rpc: dial tcp 145.79.227.5:4222: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054117039 (542h3m8.925200074s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054285930 (542h3m8.925368965s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: read tcp 188.166.225.170:37674->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054293886 (542h3m8.925376931s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054329273 (542h3m8.925412308s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:46006->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054343089 (542h3m8.925426134s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054372255 (542h3m8.92545529s) [seg_pos={0 0}, upload=475340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:32:09.054466905 (542h3m8.92554995s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:09.054473147 (542h3m8.925556182s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:09.348157750 (542h3m9.219240805s) [upload=475340]: Done uploading segments... Dec 28 05:32:09.348180353 (542h3m9.219263388s) [upload=475340]: Upload finished. err: <nil> Duration: 6.206959387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.............................X.........X...X] duration: 1m0.003168839s trace: https://jaeger.dev.storj.io/trace/13fa7ae81fb54395 FAIL (2025-12-28 05:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece... Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments... Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil> Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces. Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments... Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil> Duration: 1m0.003180442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X..] duration: 5.000873864s trace: https://jaeger.dev.storj.io/trace/74b7dbd71910e0e2 FAIL (2025-12-28 05:32:41 UTC) orbiter warning: context deadline exceeded Duration: 5.000975879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.XX....X] duration: 30.00232521s trace: https://jaeger.dev.storj.io/trace/4c1d92962d1bb6f7 FAIL (2025-12-28 05:32:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:32:04.141520669 (542h3m4.012603715s) [upload=475340]: Starting upload Dec 28 05:32:04.141623095 (542h3m4.01270614s) [upload=475340]: Done starting upload Dec 28 05:32:04.141722734 (542h3m4.01280578s) [upload=475340]: Uploading segments... Dec 28 05:32:04.141929738 (542h3m4.013012783s) [upload=475340]: Got next segment. Inline: false Dec 28 05:32:04.141974574 (542h3m4.013057619s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:32:04.141982298 (542h3m4.013065343s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:32:04.388024523 (542h3m4.259107578s) [seg_pos={0 0}, upload=475340]: Begin upload segment... Dec 28 05:32:04.388096250 (542h3m4.259179295s) [seg_pos={0 0}, upload=475340]: Done begin upload segment. Dec 28 05:32:04.388109795 (542h3m4.25919284s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388140824 (542h3m4.259223869s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.388700881 (542h3m4.259783936s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389402006 (542h3m4.260485051s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389712988 (542h3m4.260796033s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.389857513 (542h3m4.260940558s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390184426 (542h3m4.261267471s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390278695 (542h3m4.26136173s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390689687 (542h3m4.261772733s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.390799647 (542h3m4.261882692s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391131820 (542h3m4.262214865s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391155024 (542h3m4.262238069s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391510411 (542h3m4.262593446s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.391985545 (542h3m4.26306859s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392440200 (542h3m4.263523246s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:04.392875239 (542h3m4.263958284s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Uploading piece... Dec 28 05:32:06.306196329 (542h3m6.177279364s) [upload=475340]: Next returned nil segment Dec 28 05:32:06.306210625 (542h3m6.17729367s) [upload=475340]: Waiting for error group managing segments... Dec 28 05:32:06.663243664 (542h3m6.534326709s) [noise=true , address=77.90.31.19:4842 , piece_id=TB54A6426NL46W4YGXJOB4O2IJT2F5NT22JFB373SQHA2E53OYNQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.701106650 (542h3m6.572189686s) [noise=true , address=45.91.116.5:4298 , piece_id=6W2UHRPZEMB6FW7DJHWUWJCGWHYMZNNEFKB66VP6UTBL3AZBSBGA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.729004512 (542h3m6.600087558s) [noise=true , address=45.91.116.6:4976 , piece_id=BDLCRS3JHEUPQA37O4YB6TBACDHLUPWVBXOPFBMPERUQLDMBMHFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.795888771 (542h3m6.666971816s) [noise=true , address=77.90.31.18:4805 , piece_id=U7VTET5ED2V534NGJPE6DMYKP2YR2JX55K53KO4NIH5RBUHT4FFA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:06.925031540 (542h3m6.796114575s) [noise=true , address=45.91.116.9:4264 , piece_id=IWR4LENX2AZ6ZL56PLPDA6UVZ3U33LMOUPVTJV6WFU3I2Z3AR32A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.137489875 (542h3m7.00857292s) [noise=true , address=145.79.227.17:4972 , piece_id=OXK22VJUVGOYFAWR2RFKTCC5QICDRKV6R2BQ3EGBL7LYVHWJWO4Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.247045706 (542h3m7.118128751s) [noise=true , address=145.79.227.16:4912 , piece_id=7GCKCLU63ESTTFCTTEKG3WVKPPBVH36UGCATL7WM4FB724D2VSFQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.421350219 (542h3m7.292433264s) [noise=true , address=145.79.227.15:4875 , piece_id=4YFZY5MVBZBQGUUKTORVEMBJAGOPMED5YAZIMTHAC7CF64BARQLQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:07.474374597 (542h3m7.345457632s) [noise=true , address=145.79.227.41:4086 , piece_id=FSSXNHP4SAVEGFNKBSSLRNGGEN7NKBE2OKFV5JYAZ4RZTUIDIIHQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.046066699 (542h3m7.917149744s) [noise=true , address=145.79.227.10:4403 , piece_id=VLZWZSY2DQFF33PO7K5YFVMUR3NILUISP6VEP43HRPLOIXEDU2FA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.381184037 (542h3m8.252267092s) [noise=true , address=145.79.227.3:4150 , piece_id=LHQ5YGEJYEMN2EVSYQCDHXUMUW52Z2ZHFC73ETZMJNJYS5NIPF3Q, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:08.503852607 (542h3m8.374935652s) [noise=true , address=145.79.227.52:4445 , piece_id=NPY5Z73Q47YEMUY4MMOBZSK4AHFNUBSCEYDT4E2M6AWOMLDPQYPQ, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053690045 (542h3m8.92477308s) [noise=true , address=145.79.227.49:4338 , piece_id=LNH4DS5G6QAZNYV7K3KWEOSXYYI6JS7GACEIHI5WLXP7IXVM33PA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: <nil> Dec 28 05:32:09.053709042 (542h3m8.924792077s) [seg_pos={0 0}, upload=475340]: Segment reached optimal threshold of 13 pieces. Dec 28 05:32:09.054106398 (542h3m8.925189443s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: rpc: dial tcp 145.79.227.5:4222: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054117039 (542h3m8.925200074s) [noise=true , address=145.79.227.5:4222 , piece_id=2UTLTXH3AIEIAYZ3WSN45XRTPUEHJMQUGTDXMPRJXO2OCRWXVK5A, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054285930 (542h3m8.925368965s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: read tcp 188.166.225.170:37674->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054293886 (542h3m8.925376931s) [noise=true , address=145.79.227.50:4373 , piece_id=POAP2WG6WUYBM53I7VTUPQJYUIIGAPED2M5NR3A4N2BYN576DJZA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054329273 (542h3m8.925412308s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:46006->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:32:09.054343089 (542h3m8.925426134s) [noise=true , address=145.79.227.33:4869 , piece_id=X6SX25H3B2EI7BQWRI6ZVD2UGNMVM273DMVWTAD5TBIW42AXKXEA, seg_pos={0 0}, upload=475340]: Long tail cancellation due to optimal threshold Dec 28 05:32:09.054372255 (542h3m8.92545529s) [seg_pos={0 0}, upload=475340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:32:09.054466905 (542h3m8.92554995s) [upload=475340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:32:09.054473147 (542h3m8.925556182s) [upload=475340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:32:09.348157750 (542h3m9.219240805s) [upload=475340]: Done uploading segments... Dec 28 05:32:09.348180353 (542h3m9.219263388s) [upload=475340]: Upload finished. err: <nil> Duration: 30.002344495s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.072780798s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:32:14 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.072807399s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.............................X.........X...] duration: 40.709091281s trace: https://jaeger.dev.storj.io/trace/175f705d7fffd456 FAIL (2025-12-28 05:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Dec 28 05:28:47.654440633 (541h59m47.525523669s) [noise=true , address=77.90.31.11:4576 , piece_id=OYMKTB3Q34LFKXYCTG3S7HKMGAUNC3VL6RBKI2EW6QEYD6TCXUWA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.655065953 (541h59m47.526148998s) [noise=true , address=77.90.31.10:4552 , piece_id=XAXIQDGZNFLSQ6NFKZ5LOEUF3QQ3R46DFO5MSYBML7GMQNJLYX4Q, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.655295891 (541h59m47.526378937s) [noise=true , address=77.90.31.16:4745 , piece_id=FAWEC7DWQBHWMPRIDZWYYR52D5TDZOIWIEUB5AE7TS2TQ6XQWY5A, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.655683800 (541h59m47.526766845s) [noise=true , address=45.91.116.8:4050 , piece_id=L7VC22CK3LAETTFRKKI43WC5U6ITY5WWNYX4CDGV3DLQV7SEHHYA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.656055829 (541h59m47.527138874s) [noise=true , address=77.90.31.12:4618 , piece_id=P6GQDBMTSYCGQ7P2SIYNQGSMZI3VMONN6MQG6HXE4Z42T3GUDTQQ, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.656422807 (541h59m47.527505852s) [noise=true , address=77.90.31.14:4687 , piece_id=KFVAKO3A7AL5RHNCFYKQLGXBDPBF4BGW3ASLW47YUD2FVGVX5JHA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.656789636 (541h59m47.527872671s) [noise=true , address=145.79.227.40:4054 , piece_id=VOOOMAOMHT4BZBDDGOJO7VKTS2VENTTGLRM6QA7SYE7DT2ATFE2A, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.657159921 (541h59m47.528242956s) [noise=true , address=64.49.13.169:4710 , piece_id=BFOWH4JDLOMZ5CBKMYFK3MBSSYODFX4GZHCRDPZ5N7TFTMWPQTYA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.657523824 (541h59m47.528606869s) [noise=true , address=145.79.227.44:4211 , piece_id=ENXH77JLVYPQQJMV6R6T2WEQOVAHDDDIL7RDNVCZ5XXX4ALMJ7IA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.657933384 (541h59m47.529016429s) [noise=true , address=145.79.227.49:4348 , piece_id=DZBPGCMY3XYGOTUSFKPLHY3UEIXRJQEZYRNACRADYM5EN5YP6FYA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.658096615 (541h59m47.52917965s) [noise=true , address=77.90.31.15:4704 , piece_id=A6JLGTFICGXQADOCLLYIG6E6YMU4GOZJWCDV4XWXS4BG2SILJAKQ, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.658313628 (541h59m47.529396673s) [noise=true , address=77.90.31.18:4824 , piece_id=L5L66P5PFGQGMLSHPPEWQ5GVQU47AHURG6U7SU2FH72VWWEQG4JQ, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.658499261 (541h59m47.529582296s) [noise=true , address=77.90.31.19:4868 , piece_id=XGAFFAJR2PMGDULPIHFSO6ZAJBOSEKN5HD5ZREQ7A3R46TS6S7WQ, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.658716555 (541h59m47.5297996s) [noise=true , address=145.79.227.35:4486 , piece_id=ZU5DZSIWJIU76HBG3QXYLKH7QQZ6TJS5W6K6CV3RY4ZOSKAHW56A, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.658928959 (541h59m47.530011994s) [noise=true , address=145.79.227.34:4919 , piece_id=TD6IGC5W25UXUG7BURGYVGT36Y5V4K26G5VRTRJFHY3DBNPIUGGA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.659114834 (541h59m47.530197879s) [noise=true , address=64.49.13.169:4709 , piece_id=7THGKM63OJFEEJUWFC6Q6HLERQSEPPSYKUF7T3UWEGPVOEAYSMGA, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:47.659197150 (541h59m47.530280185s) [noise=true , address=77.90.31.13:4659 , piece_id=TKYFXXWU755IFEBHE5PECKMRKG3BLPXURRNMJ6W7K4KIWHK4LW2Q, seg_pos={0 0}, upload=475290]: Uploading piece... Dec 28 05:28:49.149024041 (541h59m49.020107086s) [upload=475290]: Next returned nil segment Dec 28 05:28:49.149053196 (541h59m49.020136242s) [upload=475290]: Waiting for error group managing segments... Dec 28 05:28:49.550894727 (541h59m49.421977762s) [noise=true , address=145.79.227.49:4348 , piece_id=DZBPGCMY3XYGOTUSFKPLHY3UEIXRJQEZYRNACRADYM5EN5YP6FYA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.797639258 (541h59m49.668722303s) [noise=true , address=77.90.31.16:4745 , piece_id=FAWEC7DWQBHWMPRIDZWYYR52D5TDZOIWIEUB5AE7TS2TQ6XQWY5A, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.804134476 (541h59m49.675217511s) [noise=true , address=77.90.31.14:4687 , piece_id=KFVAKO3A7AL5RHNCFYKQLGXBDPBF4BGW3ASLW47YUD2FVGVX5JHA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.835886487 (541h59m49.706969532s) [noise=true , address=145.79.227.33:4894 , piece_id=SNXL2YIN7B4FCKW7EABJCMVMHI2CZP7XGKUP663PY3FVGOBWRNIQ, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.856243820 (541h59m49.727326855s) [noise=true , address=77.90.31.13:4659 , piece_id=TKYFXXWU755IFEBHE5PECKMRKG3BLPXURRNMJ6W7K4KIWHK4LW2Q, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.856711921 (541h59m49.727794976s) [noise=true , address=45.91.116.8:4050 , piece_id=L7VC22CK3LAETTFRKKI43WC5U6ITY5WWNYX4CDGV3DLQV7SEHHYA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:49.866666366 (541h59m49.737749421s) [noise=true , address=77.90.31.19:4868 , piece_id=XGAFFAJR2PMGDULPIHFSO6ZAJBOSEKN5HD5ZREQ7A3R46TS6S7WQ, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:50.055129090 (541h59m49.926212125s) [noise=true , address=77.90.31.11:4576 , piece_id=OYMKTB3Q34LFKXYCTG3S7HKMGAUNC3VL6RBKI2EW6QEYD6TCXUWA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:50.159053438 (541h59m50.030136473s) [noise=true , address=77.90.31.15:4704 , piece_id=A6JLGTFICGXQADOCLLYIG6E6YMU4GOZJWCDV4XWXS4BG2SILJAKQ, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:50.479285521 (541h59m50.350368556s) [noise=true , address=145.79.227.44:4211 , piece_id=ENXH77JLVYPQQJMV6R6T2WEQOVAHDDDIL7RDNVCZ5XXX4ALMJ7IA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:50.511575346 (541h59m50.382658381s) [noise=true , address=77.90.31.10:4552 , piece_id=XAXIQDGZNFLSQ6NFKZ5LOEUF3QQ3R46DFO5MSYBML7GMQNJLYX4Q, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:50.652781716 (541h59m50.523864751s) [noise=true , address=77.90.31.18:4824 , piece_id=L5L66P5PFGQGMLSHPPEWQ5GVQU47AHURG6U7SU2FH72VWWEQG4JQ, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:28:53.197419804 (541h59m53.068502839s) [noise=true , address=77.90.31.12:4618 , piece_id=P6GQDBMTSYCGQ7P2SIYNQGSMZI3VMONN6MQG6HXE4Z42T3GUDTQQ, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:29:03.107357650 (542h0m2.978440695s) [noise=true , address=64.49.13.169:4709 , piece_id=7THGKM63OJFEEJUWFC6Q6HLERQSEPPSYKUF7T3UWEGPVOEAYSMGA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: <nil> Dec 28 05:29:03.107384462 (542h0m2.978467517s) [seg_pos={0 0}, upload=475290]: Segment reached optimal threshold of 14 pieces. Dec 28 05:29:03.107573081 (542h0m2.978656126s) [noise=true , address=145.79.227.40:4054 , piece_id=VOOOMAOMHT4BZBDDGOJO7VKTS2VENTTGLRM6QA7SYE7DT2ATFE2A, seg_pos={0 0}, upload=475290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 28 05:29:03.107582759 (542h0m2.978665804s) [noise=true , address=145.79.227.40:4054 , piece_id=VOOOMAOMHT4BZBDDGOJO7VKTS2VENTTGLRM6QA7SYE7DT2ATFE2A, seg_pos={0 0}, upload=475290]: Long tail cancellation due to optimal threshold Dec 28 05:29:03.107790475 (542h0m2.97887353s) [noise=true , address=145.79.227.35:4486 , piece_id=ZU5DZSIWJIU76HBG3QXYLKH7QQZ6TJS5W6K6CV3RY4ZOSKAHW56A, seg_pos={0 0}, upload=475290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 28 05:29:03.107799702 (542h0m2.978882737s) [noise=true , address=145.79.227.35:4486 , piece_id=ZU5DZSIWJIU76HBG3QXYLKH7QQZ6TJS5W6K6CV3RY4ZOSKAHW56A, seg_pos={0 0}, upload=475290]: Long tail cancellation due to optimal threshold Dec 28 05:29:03.108070218 (542h0m2.979153263s) [noise=true , address=64.49.13.169:4710 , piece_id=BFOWH4JDLOMZ5CBKMYFK3MBSSYODFX4GZHCRDPZ5N7TFTMWPQTYA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: expected piece hash; context canceled; EOF Dec 28 05:29:03.108095967 (542h0m2.979179012s) [noise=true , address=64.49.13.169:4710 , piece_id=BFOWH4JDLOMZ5CBKMYFK3MBSSYODFX4GZHCRDPZ5N7TFTMWPQTYA, seg_pos={0 0}, upload=475290]: Long tail cancellation due to optimal threshold Dec 28 05:29:03.108367844 (542h0m2.979450899s) [noise=true , address=145.79.227.34:4919 , piece_id=TD6IGC5W25UXUG7BURGYVGT36Y5V4K26G5VRTRJFHY3DBNPIUGGA, seg_pos={0 0}, upload=475290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 28 05:29:03.108376792 (542h0m2.979459837s) [noise=true , address=145.79.227.34:4919 , piece_id=TD6IGC5W25UXUG7BURGYVGT36Y5V4K26G5VRTRJFHY3DBNPIUGGA, seg_pos={0 0}, upload=475290]: Long tail cancellation due to optimal threshold Dec 28 05:29:03.108425534 (542h0m2.979508579s) [seg_pos={0 0}, upload=475290]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:29:03.108562024 (542h0m2.979645079s) [upload=475290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:29:03.108595798 (542h0m2.979678863s) [upload=475290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:29:03.407909219 (542h0m3.278992265s) [upload=475290]: Done uploading segments... Dec 28 05:29:03.407938174 (542h0m3.279021219s) [upload=475290]: Upload finished. err: <nil> Duration: 40.709107742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.X.] duration: 4.943167598s trace: https://jaeger.dev.storj.io/trace/74b7dbd71910e0e2 FAIL (2025-12-28 05:31:43 UTC) orbiter warning: context deadline exceeded Duration: 4.943197565s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)