Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.X.......X.X.XXX.X.....X.......X...X....................] duration: 6.073940814s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:07:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.073969298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.X.......X.X.XXX.X.....X.......X...X...................] duration: 5.757732912s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.757763561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.546197432s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:07:00 UTC) orbiter warning: context deadline exceeded Duration: 3.546212921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.X.......X.X.XXX.X.....X.......X...X..................] duration: 6.402838943s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:06:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.402866295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.51236572s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:05:59 UTC) orbiter warning: context deadline exceeded Duration: 3.512394115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X.X.......X.X.XXX.X.....X.......X...X.................] duration: 11.380252064s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:05:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 11.380284575s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.457560954s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:04:53 UTC) orbiter warning: context deadline exceeded Duration: 3.457624655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X.X.......X.X.XXX.X.....X.......X...X................] duration: 5.71807758s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.718127115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.43265594s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:04:10 UTC) orbiter warning: context deadline exceeded Duration: 3.432696748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.483216076s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:03:47 UTC) orbiter warning: context deadline exceeded Duration: 3.483292642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.......X.X.......X.X.XXX.X.....X.......X...X...............] duration: 5.98513909s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.985168436s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.510082494s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:03:28 UTC) orbiter warning: context deadline exceeded Duration: 3.510110566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......X.X.......X.X.XXX.X.....X.......X...X..............] duration: 5.962099321s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.962129519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 4.898483535s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:02:49 UTC) orbiter warning: context deadline exceeded Duration: 4.898636034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 5.001242385s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:02:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001267742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.......X.X.......X.X.XXX.X.....X.......X...X.............] duration: 5.741699422s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.741746292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.558027479s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:01:46 UTC) orbiter warning: context deadline exceeded Duration: 3.558057096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X.X.......X.X.XXX.X.....X.......X...X............] duration: 5.907944277s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.907970778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.370085637s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:00:49 UTC) orbiter warning: context deadline exceeded Duration: 3.370115182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.......X.X.......X.X.XXX.X.....X.......X...X...........] duration: 6.095428702s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 07:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 6.095457638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.972349197s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:00:31 UTC) orbiter warning: context deadline exceeded Duration: 3.972438999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.......X.X.......X.X.XXX.X.....X.......X...X..........] duration: 5.755715498s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.755751737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.533887641s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 06:59:30 UTC) orbiter warning: context deadline exceeded Duration: 3.533914953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.......X.X.......X.X.XXX.X.....X.......X...X.........] duration: 5.75011879s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:58:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.750138509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 4.966710029s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 06:58:45 UTC) orbiter warning: context deadline exceeded Duration: 4.96675234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.......X.X.......X.X.XXX.X.....X.......X...X........] duration: 5.972001255s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.972025962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.503021729s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 06:58:21 UTC) orbiter warning: context deadline exceeded Duration: 3.503050945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.........................................................X.] duration: 5.002029585s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 06:58:03 UTC) orbiter warning: context deadline exceeded Duration: 5.002093917s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.........................................................X] duration: 5.001169676s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 06:57:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001195315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X.......X.X.......X.X.XXX.X.....X.......X...X.......] duration: 5.748018614s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.74804905s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 5.000419198s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:56:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000444106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X.......X.X.......X.X.XXX.X.....X.......X...X......] duration: 5.736824958s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.736848734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X.......X.X.......X.X.XXX.X.....X.......X...X.....] duration: 5.918882708s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.918907647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..........X.......X.X.......X.X.XXX.X.....X.......X...X....] duration: 5.806585774s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:55:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.806616364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.160707853s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:55:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.160737379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..........X.......X.X.......X.X.XXX.X.....X.......X...X...] duration: 5.917051611s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:54:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.917076269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 4.789034516s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:54:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.789061667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..........X.......X.X.......X.X.XXX.X.....X.......X...X..] duration: 5.929798044s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 5.929829944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.508555957s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:53:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508603867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..........X.......X.X.......X.X.XXX.X.....X.......X...X.] duration: 7.021093146s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:53:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 7.021123084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X..........X.......X.X.......X.X.XXX.X.....X.......X...X] duration: 30.003144469s trace: https://jaeger.dev.storj.io/trace/714399c058118860 FAIL (2025-12-26 06:52:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments... Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment... Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment. Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece... Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments... Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014 , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302 , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293 , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329 , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339 , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086 , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266 , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445 , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006 , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565 , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974 , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226 , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519 , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil> Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces. Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937 , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590 , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156 , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments... Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil> Duration: 30.003175879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.672808742s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:52:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.67282926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.828287553s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:51:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.828313994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..........X.......X.X.......X.X.XXX.X.....X.......X...] duration: 5.985187849s trace: https://jaeger.dev.storj.io/trace/18840c7c7f742458 FAIL (2025-12-26 06:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded ======LOGS====== : Done starting upload Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments... Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment... Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment. Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments... Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces. Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments... Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil> Duration: 5.985218195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 4.414944678s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:51:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.414974504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.547472492s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:50:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547675269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X..........X.......X.X.......X.X.XXX.X.....X.......X..] duration: 5.760977442s trace: https://jaeger.dev.storj.io/trace/18840c7c7f742458 FAIL (2025-12-26 06:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded ======LOGS====== : Done starting upload Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments... Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment... Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment. Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments... Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces. Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments... Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil> Duration: 5.76100712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 4.887092125s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:50:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.887122783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X..........X.......X.X.......X.X.XXX.X.....X.......X.] duration: 6.026157649s trace: https://jaeger.dev.storj.io/trace/18840c7c7f742458 FAIL (2025-12-26 06:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded ======LOGS====== : Done starting upload Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments... Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment... Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment. Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece... Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments... Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566 , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013 , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617 , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257 , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403 , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519 , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266 , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941 , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873 , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446 , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082 , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010 , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil> Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces. Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299 , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554 , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments... Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil> Duration: 6.026188248s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.37126895s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:49:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37129523s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)