Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 326879 reports (Page 61 of 6538)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
13 out of 60 is failed (threshold=3) [...X..........X.......X.X.......X.X.XXX.X.....X.......X...X.]
duration: 
7.021093146s
trace: 
https://jaeger.dev.storj.io/trace/714399c058118860
FAIL (2025-12-26 06:53:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload
Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload
Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments...
Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false
Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment...
Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment.
Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086   , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329    , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565   , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226    , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014   , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519     , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266   , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445   , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974     , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339   , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302   , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006   , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293    , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment
Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments...
Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014   , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302   , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293    , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329    , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339   , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086   , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266   , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445   , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006   , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565   , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974     , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226    , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519     , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection
Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments...
Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil>
Duration: 7.021123084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
13 out of 60 is failed (threshold=3) [....X..........X.......X.X.......X.X.XXX.X.....X.......X...X]
duration: 
30.003144469s
trace: 
https://jaeger.dev.storj.io/trace/714399c058118860
FAIL (2025-12-26 06:52:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:52:02.598083016 (495h23m2.469166061s) [upload=434800]: Starting upload
Dec 26 06:52:02.598143491 (495h23m2.469226536s) [upload=434800]: Done starting upload
Dec 26 06:52:02.598310309 (495h23m2.469393364s) [upload=434800]: Uploading segments...
Dec 26 06:52:02.598411130 (495h23m2.469494175s) [upload=434800]: Got next segment. Inline: false
Dec 26 06:52:02.598474170 (495h23m2.469557225s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:52:02.598537731 (495h23m2.469620777s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:52:02.846892300 (495h23m2.717975355s) [seg_pos={0 0}, upload=434800]: Begin upload segment...
Dec 26 06:52:02.846952725 (495h23m2.71803576s) [seg_pos={0 0}, upload=434800]: Done begin upload segment.
Dec 26 06:52:02.846988654 (495h23m2.718071699s) [noise=true , address=145.79.227.41:4086   , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847104365 (495h23m2.71818741s) [noise=true , address=145.79.227.8:4329    , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847810519 (495h23m2.718893564s) [noise=true , address=145.79.227.37:4565   , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.847872958 (495h23m2.718956003s) [noise=true , address=45.91.116.12:4226    , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848268040 (495h23m2.719351085s) [noise=true , address=145.79.227.39:4014   , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848442883 (495h23m2.719525918s) [noise=true , address=77.90.31.10:4519     , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.848860679 (495h23m2.719943724s) [noise=true , address=145.79.227.47:4266   , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849153566 (495h23m2.720236611s) [noise=true , address=145.79.227.52:4445   , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849300376 (495h23m2.720383411s) [noise=true , address=45.91.116.6:4974     , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849714234 (495h23m2.720797279s) [noise=true , address=145.79.227.49:4339   , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.849759410 (495h23m2.720842455s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850150444 (495h23m2.72123349s) [noise=true , address=145.79.227.48:4302   , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850247640 (495h23m2.721330685s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850683269 (495h23m2.721766344s) [noise=true , address=145.79.227.18:4006   , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.850834427 (495h23m2.721917472s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:02.851277140 (495h23m2.722360175s) [noise=true , address=145.79.227.7:4293    , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Uploading piece...
Dec 26 06:52:04.575832387 (495h23m4.446915432s) [upload=434800]: Next returned nil segment
Dec 26 06:52:04.575846244 (495h23m4.446929289s) [upload=434800]: Waiting for error group managing segments...
Dec 26 06:52:04.863184836 (495h23m4.734267871s) [noise=true , address=145.79.227.39:4014   , piece_id=LGT3G4LZNRMVJIBQMQT7I2V6WUWERY24WIVODDFIKY5BYM6FYD7Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.864687276 (495h23m4.735770321s) [noise=true , address=145.79.227.48:4302   , piece_id=6JOTEMHTXN4GJQZA7QUZQ2SUDJZSR45MT4HPSMWW76OG2ZH3KEEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.864846299 (495h23m4.735929334s) [noise=true , address=145.79.227.7:4293    , piece_id=QLGCAJN3O3L2DSE526TTZQ6WZNSEDPMMEJWCLAVKLBRMQY4NX7NQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.865985308 (495h23m4.737068343s) [noise=true , address=145.79.227.8:4329    , piece_id=YHJFDADNAWLPIMAPRZ3XMIQA7AWWEEAZXVFNUN5BVVTOPQSLZPLQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.866063536 (495h23m4.737146571s) [noise=true , address=145.79.227.49:4339   , piece_id=OJ3HOBYOKFKK74MBSOZXIGNH7Z4S7HX5VMMB535MEYAQZQW3GK4A, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.870226594 (495h23m4.741309629s) [noise=true , address=145.79.227.41:4086   , piece_id=KSVBD2QULXBCF5OIALMYBEQQ2C6XIJVA3XDUBFR2UMFXOG3XIWEQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.877240058 (495h23m4.748323103s) [noise=true , address=145.79.227.47:4266   , piece_id=DXWVRRZGRTLW44E5XLMDOIGZVAEEWVOWICSHCFSPTNKENWBUQ4UQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.877479184 (495h23m4.748562229s) [noise=true , address=145.79.227.52:4445   , piece_id=DYITKOOL7K7GR6LI6UOMWXMUCLAOZ5WKIOP56MKJCI7SBWJS4VMQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.879892991 (495h23m4.750976036s) [noise=true , address=145.79.227.18:4006   , piece_id=4FNELR6MDXPHY3CX5RR5W4CDA5Q7P7Z5SHRZVJ4IE74NERFZ544Q, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:04.888779071 (495h23m4.759862106s) [noise=true , address=145.79.227.37:4565   , piece_id=WM55YTFRGREXD6I46CX2UZNEFSNYVDBSBNZC6X3RZKXWKHNO43NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.125416589 (495h23m4.996499634s) [noise=true , address=45.91.116.6:4974     , piece_id=CB4Q3432TQQ4WJ25OEINRIHPPBV6FZ5MCCLZBLRENXYRQS5I6KKQ, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.177172210 (495h23m5.048255245s) [noise=true , address=45.91.116.12:4226    , piece_id=GJGQTEZUMS7Q36AXBQPS3B2P3V5SBIRXQMBJZWQHK52EOH5MLTTA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.195127350 (495h23m5.066210385s) [noise=true , address=77.90.31.10:4519     , piece_id=GYZEROWQKVHK5NHDRRZ6GSVYZIYLKPUKSK7O6RFWOYECCTW5Q3FA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: <nil>
Dec 26 06:52:05.195151596 (495h23m5.066234631s) [seg_pos={0 0}, upload=434800]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:52:05.195404718 (495h23m5.066487753s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 26 06:52:05.195415799 (495h23m5.066498844s) [noise=true , address=45.91.116.4:4937     , piece_id=NV6ZM4L6JCKYXUF7ZDEPA7N5RUIQ7BT4NEG4WESOKOEELJAC54OA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195476214 (495h23m5.066559249s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 06:52:05.195492325 (495h23m5.06657537s) [noise=true , address=77.90.31.12:4590     , piece_id=SISWPWSVVCU4H6DPHFRXI4QER37XHC4AKRAYEOSAGRRQ2PPCX4NA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195555996 (495h23m5.066639031s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38564->45.91.116.13:4156: use of closed network connection
Dec 26 06:52:05.195563000 (495h23m5.066646035s) [noise=true , address=45.91.116.13:4156    , piece_id=C6XZHGTKQOCJVAXVPAWBQ73MFJRD5AJPWB2WT37QTIYD24OVA2PA, seg_pos={0 0}, upload=434800]: Long tail cancellation due to optimal threshold
Dec 26 06:52:05.195592736 (495h23m5.066675771s) [seg_pos={0 0}, upload=434800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:52:05.195663891 (495h23m5.066746936s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:52:05.195668501 (495h23m5.066751536s) [upload=434800]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:52:05.482722289 (495h23m5.353805335s) [upload=434800]: Done uploading segments...
Dec 26 06:52:05.482745284 (495h23m5.353828319s) [upload=434800]: Upload finished. err: <nil>
Duration: 30.003175879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......X.....................................................]
duration: 
4.672808742s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:52:31 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.67282926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......X....................................................]
duration: 
4.828287553s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:51:33 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.828313994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
12 out of 60 is failed (threshold=3) [.....X..........X.......X.X.......X.X.XXX.X.....X.......X...]
duration: 
5.985187849s
trace: 
https://jaeger.dev.storj.io/trace/18840c7c7f742458
FAIL (2025-12-26 06:51:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded
======LOGS======
: Done starting upload
Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments...
Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false
Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment...
Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment.
Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment
Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments...
Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection
Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled
Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments...
Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil>
Duration: 5.985218195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........X...................................................]
duration: 
4.414944678s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:51:05 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.414974504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........X..................................................]
duration: 
3.547472492s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:50:21 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.547675269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
12 out of 60 is failed (threshold=3) [......X..........X.......X.X.......X.X.XXX.X.....X.......X..]
duration: 
5.760977442s
trace: 
https://jaeger.dev.storj.io/trace/18840c7c7f742458
FAIL (2025-12-26 06:50:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded
======LOGS======
: Done starting upload
Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments...
Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false
Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment...
Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment.
Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment
Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments...
Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection
Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled
Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments...
Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil>
Duration: 5.76100712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........X.................................................]
duration: 
4.887092125s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:50:06 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.887122783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
12 out of 60 is failed (threshold=3) [.......X..........X.......X.X.......X.X.XXX.X.....X.......X.]
duration: 
6.026157649s
trace: 
https://jaeger.dev.storj.io/trace/18840c7c7f742458
FAIL (2025-12-26 06:49:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded
======LOGS======
: Done starting upload
Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments...
Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false
Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment...
Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment.
Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment
Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments...
Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection
Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled
Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments...
Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil>
Duration: 6.026188248s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........X................................................]
duration: 
3.37126895s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:49:30 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.37129523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
12 out of 60 is failed (threshold=3) [........X..........X.......X.X.......X.X.XXX.X.....X.......X]
duration: 
30.00248211s
trace: 
https://jaeger.dev.storj.io/trace/18840c7c7f742458
FAIL (2025-12-26 06:49:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA) context deadline exceeded
======LOGS======
: Done starting upload
Dec 26 06:48:49.307694737 (495h19m49.178777772s) [upload=434755]: Uploading segments...
Dec 26 06:48:49.307865172 (495h19m49.178948207s) [upload=434755]: Got next segment. Inline: false
Dec 26 06:48:49.307910528 (495h19m49.178993573s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:48:49.307916088 (495h19m49.178999123s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:48:49.555920341 (495h19m49.427003376s) [seg_pos={0 0}, upload=434755]: Begin upload segment...
Dec 26 06:48:49.555987539 (495h19m49.427070574s) [seg_pos={0 0}, upload=434755]: Done begin upload segment.
Dec 26 06:48:49.556027454 (495h19m49.427110489s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556037945 (495h19m49.42712098s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556094081 (495h19m49.427177126s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556532176 (495h19m49.427615221s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.556917430 (495h19m49.428000475s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557312642 (495h19m49.428395687s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.557734244 (495h19m49.428817279s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558337063 (495h19m49.429420098s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.558564626 (495h19m49.429647671s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559056202 (495h19m49.430139247s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.559651165 (495h19m49.4307342s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560491526 (495h19m49.431574561s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.560544396 (495h19m49.431627431s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561056461 (495h19m49.432139496s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561221466 (495h19m49.432304511s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:49.561974859 (495h19m49.433057895s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Uploading piece...
Dec 26 06:48:51.270840692 (495h19m51.141923737s) [upload=434755]: Next returned nil segment
Dec 26 06:48:51.270858325 (495h19m51.14194138s) [upload=434755]: Waiting for error group managing segments...
Dec 26 06:48:51.555282748 (495h19m51.426365783s) [noise=true , address=145.79.227.37:4566   , piece_id=WFIA2LKIJQ5QRCFDHUFPJILR3YFLPQMTAJOAGAXAYDHBL23B7WHQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.559609676 (495h19m51.430692711s) [noise=true , address=145.79.227.39:4013   , piece_id=NYSRGI7SDDU3HKN3TBPGILIAFW3FMAOIPQC67Y2QWAKLZ7SPBPDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.561587503 (495h19m51.432670548s) [noise=true , address=145.79.227.25:4617   , piece_id=4XREMO3KF44RLTV65EFHBHDRYAQPYOQROV5OR5TWFWWB3KJK7ZLQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.565147683 (495h19m51.436230728s) [noise=true , address=145.79.227.6:4257    , piece_id=UBZRMIUUI7H5MIWIMDKSSGT3M3D4AQ24IHMBYBUKPPZIVWZVDMMQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.569067456 (495h19m51.440150491s) [noise=true , address=145.79.227.10:4403   , piece_id=3F3LERQKHFB4TKRH3U5KIQZLKNIW2OZO6MMO62FXPP52OGJ7LCNA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.578463077 (495h19m51.449546122s) [noise=true , address=145.79.227.36:4519   , piece_id=3KL7QA7NSBHKW2THGIIWYV7AILV46GGNHDGZST52QZU3DL6JFLRA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.581793208 (495h19m51.452876254s) [noise=true , address=145.79.227.47:4266   , piece_id=IPFNU7VT2WDVQIRNICKF65RWXP33CSG5MVZRSD76OYH675I7S3FQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.598795313 (495h19m51.469878358s) [noise=true , address=145.79.227.29:4941   , piece_id=WCC3HLHST2AUPGYVZ2ZQ7KDU6LPL3XLA5XQ5WYSM3FWNRH4V4XGA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603519850 (495h19m51.474602895s) [noise=true , address=145.79.227.15:4873   , piece_id=TZ7TW625G2BJHURFXJ2OT2CX36NJBWNH54IVILWLEB6AVABD6LDA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.603937345 (495h19m51.47502038s) [noise=true , address=145.79.227.52:4446   , piece_id=EDYEUB7FXPZXHKWNZVNE5SINU6NRSJYRQJUDBGDMQ2R2BUMCQV2A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.794008662 (495h19m51.665091717s) [noise=true , address=45.91.116.10:4082    , piece_id=VL6HLUBCKOPQT2WANLRTDWOQ4U3GJ5VWZUBDPOGXEFASLEJQKPEQ, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.821726412 (495h19m51.692809457s) [noise=true , address=45.91.116.8:4046     , piece_id=Y4QVQJEUAOG6RL6YEQ4IXGE37C4IPBMW57ZEOG2PP3GZIR3YS46A, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929389971 (495h19m51.800473016s) [noise=true , address=45.91.116.7:4010     , piece_id=CHHMYBCJ6JLXLJTXBSSTFBIR2FNSQANDIAJ6LZMHSTYF3T5T6KAA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: <nil>
Dec 26 06:48:51.929409719 (495h19m51.800492754s) [seg_pos={0 0}, upload=434755]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:48:51.929586706 (495h19m51.800669741s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->45.91.116.5:4299: use of closed network connection
Dec 26 06:48:51.929592908 (495h19m51.800675943s) [noise=true , address=45.91.116.5:4299     , piece_id=P2PUWDEKAT46L5ZDBRC75EWMEKXVKDH43R43PH4F5AKIWNAEOW4Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.929679774 (495h19m51.800762809s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:38022->77.90.31.11:4554: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 06:48:51.929684693 (495h19m51.800767738s) [noise=true , address=77.90.31.11:4554     , piece_id=YFEF7LO7PHL3W3KXEFNWWWUKGJIG5B7S3JKPXIFD4RNIQGHU7ILA, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930388463 (495h19m51.801471508s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43584->45.91.116.9:4261: use of closed network connection; context canceled
Dec 26 06:48:51.930407148 (495h19m51.801490183s) [noise=true , address=45.91.116.9:4261     , piece_id=MQ2THBEDDZWXVSSPKW4YQWNFEIHMWAPJN2FPWKCW2UN2HRU3GV2Q, seg_pos={0 0}, upload=434755]: Long tail cancellation due to optimal threshold
Dec 26 06:48:51.930436234 (495h19m51.801519279s) [seg_pos={0 0}, upload=434755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:48:51.930652355 (495h19m51.80173541s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:48:51.930659690 (495h19m51.801742745s) [upload=434755]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:48:52.230999878 (495h19m52.102082934s) [upload=434755]: Done uploading segments...
Dec 26 06:48:52.231076634 (495h19m52.102159679s) [upload=434755]: Upload finished. err: <nil>
Duration: 30.00249767s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............X...............................................]
duration: 
3.454225806s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:48:21 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.454253738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [.........X..........X.......X.X.......X.X.XXX.X.....X.......]
duration: 
6.114163356s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:48:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 6.11418084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [..........X..........X.......X.X.......X.X.XXX.X.....X......]
duration: 
5.882398831s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:47:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 5.882420633s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.............X..............................................]
duration: 
3.404605078s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:47:06 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.404630196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..............X.............................................]
duration: 
3.552086119s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:46:41 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.552112788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [...........X..........X.......X.X.......X.X.XXX.X.....X.....]
duration: 
5.715189135s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:46:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 5.715216486s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...............X............................................]
duration: 
3.454337287s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:45:30 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.454355631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [............X..........X.......X.X.......X.X.XXX.X.....X....]
duration: 
5.797892197s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:45:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 5.797914418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [................X...........................................]
duration: 
3.490858932s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:44:53 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.490875302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [.............X..........X.......X.X.......X.X.XXX.X.....X...]
duration: 
6.063426526s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:44:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 6.063456884s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.................X..........................................]
duration: 
4.509283944s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:44:20 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.509308751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..................X.........................................]
duration: 
3.524239092s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:44:04 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.524345405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [..............X..........X.......X.X.......X.X.XXX.X.....X..]
duration: 
26.38288621s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:43:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 26.382929182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...................X........................................]
duration: 
3.403775777s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:42:59 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.403803709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [...............X..........X.......X.X.......X.X.XXX.X.....X.]
duration: 
5.753203971s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:42:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 5.753223519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [................X..........X.......X.X.......X.X.XXX.X.....X]
duration: 
30.00429327s
trace: 
https://jaeger.dev.storj.io/trace/6902a4df5838be7e
FAIL (2025-12-26 06:42:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 06:42:01.999857277 (495h13m1.870940333s) [upload=434656]: Starting upload
Dec 26 06:42:01.999979471 (495h13m1.871062516s) [upload=434656]: Done starting upload
Dec 26 06:42:02.000354475 (495h13m1.87143754s) [upload=434656]: Uploading segments...
Dec 26 06:42:02.000410882 (495h13m1.871493927s) [upload=434656]: Got next segment. Inline: false
Dec 26 06:42:02.000451729 (495h13m1.871534784s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:42:02.000458633 (495h13m1.871541688s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:42:02.245205291 (495h13m2.116288336s) [seg_pos={0 0}, upload=434656]: Begin upload segment...
Dec 26 06:42:02.245275034 (495h13m2.116358069s) [seg_pos={0 0}, upload=434656]: Done begin upload segment.
Dec 26 06:42:02.245308939 (495h13m2.116391984s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245327965 (495h13m2.116411011s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245404451 (495h13m2.116487486s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.245950872 (495h13m2.117033907s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.246371242 (495h13m2.117454277s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.247843034 (495h13m2.118926069s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.248586471 (495h13m2.119669516s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.249789160 (495h13m2.120872195s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.251389257 (495h13m2.122472302s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.252725522 (495h13m2.123808567s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.254206422 (495h13m2.125289457s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255576320 (495h13m2.126659355s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.255770219 (495h13m2.126853254s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257187358 (495h13m2.128270403s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.257287589 (495h13m2.128370634s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:02.259064513 (495h13m2.130147558s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Uploading piece...
Dec 26 06:42:03.975620609 (495h13m3.846703654s) [upload=434656]: Next returned nil segment
Dec 26 06:42:03.975641969 (495h13m3.846725014s) [upload=434656]: Waiting for error group managing segments...
Dec 26 06:42:04.259377740 (495h13m4.130460785s) [noise=true , address=145.79.227.33:4869   , piece_id=2ZS2BMTGS2TSASVECQETTZZWFE5DXBFYBKO4WP74YCHAUNLJ6FHQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.261420019 (495h13m4.132503064s) [noise=true , address=145.79.227.13:4802   , piece_id=DLAYKGOXPHXQI2FH74YMR2TPUUXAIDMIIHGN42MRR7BAJSIWOX6A, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.262450411 (495h13m4.133533456s) [noise=true , address=145.79.227.48:4301   , piece_id=SJCM4X5CIU6GN4WSKEIFQT4EC3WKIDT7X5D4X4QVB4BZG3ZJ6TQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.264374525 (495h13m4.13545757s) [noise=true , address=145.79.227.6:4259    , piece_id=4Q2U7FC3C7THALFVWZ67GY3FASRDX2QDSXIVYTELGJO3PAKIIAPQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266745059 (495h13m4.137828094s) [noise=true , address=145.79.227.11:4732   , piece_id=KEMQSUQDGJVX6W5VLKWKQQNV2Q2BK3EN3CE4LTGCXHYGYZOH6ITA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.266987350 (495h13m4.138070395s) [noise=true , address=145.79.227.17:4971   , piece_id=WY5P5J5YNPRS43ZELA4O4FVGNCLDRNDGBXAYOAROQ6NQFQ5HFFCQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.269039889 (495h13m4.140122924s) [noise=true , address=145.79.227.31:4798   , piece_id=OBVUI7GYLCF5WDRFTFYYJIJUYDL7L64UOA2J25J3AWEPQPADBTHA, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.274945244 (495h13m4.146028279s) [noise=true , address=145.79.227.2:4113    , piece_id=OEMNAP5SGYTWHNNLLB6XRG5K6M55CXRTEWK5CGKJ7IRK5TVGJQ6Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.277233621 (495h13m4.148316666s) [noise=true , address=145.79.227.42:4123   , piece_id=XXQFHUOBPRZA32OISQ5GVEORR2SU4YMNFUGT7TRDP42CKOUQLTIQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.279053947 (495h13m4.150136993s) [noise=true , address=145.79.227.29:4943   , piece_id=EJK6ONOXQCPELTFDFBOV5JJX6MBJGDGYESD66KVQVZ4U62RE56QQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.424325209 (495h13m4.295408254s) [noise=true , address=145.79.227.52:4446   , piece_id=YOMEKUKROFFLGHRRNTUMPBM4FAA4QL4CX76W2BIMWNGXVZAMCHQQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496042730 (495h13m4.367125765s) [noise=true , address=45.91.116.13:4154    , piece_id=7N64IQ3WAEW5CE5C6LTIWYGSTFOOOUZ5TENY2SF7WLRLNA57VS4Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496328775 (495h13m4.36741182s) [noise=true , address=45.91.116.7:4012     , piece_id=BIOJDXFYLQDFI4GJNNHVKUKCGP56735ABNISJQLPFDMHAQX7ZABQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: <nil>
Dec 26 06:42:04.496352330 (495h13m4.367435365s) [seg_pos={0 0}, upload=434656]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:42:04.496546821 (495h13m4.367629866s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496553563 (495h13m4.367636608s) [noise=true , address=45.91.116.11:4120    , piece_id=7FURP7KCN2HZZU5OA2WPIQ7SNLPE7B64R4YSXTNNOV2YWJ7EFE7Q, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496599981 (495h13m4.367683026s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496609320 (495h13m4.367692355s) [noise=true , address=45.91.116.8:4048     , piece_id=NB3TR6NXB2VQQEAZSCUNPKJNDVDG4O2IE2IOHC3YMUPTZTZ7HTVQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496682158 (495h13m4.367765203s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 26 06:42:04.496688540 (495h13m4.367771585s) [noise=true , address=45.91.116.9:4263     , piece_id=C32S3MMIHT6XGKYBTABEER2Z7WTR4ECYWWBKVUMEJKHPDWLZUMXQ, seg_pos={0 0}, upload=434656]: Long tail cancellation due to optimal threshold
Dec 26 06:42:04.496706324 (495h13m4.367789359s) [seg_pos={0 0}, upload=434656]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:42:04.496793260 (495h13m4.367876305s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:42:04.496797918 (495h13m4.367880954s) [upload=434656]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:42:04.788378678 (495h13m4.659461723s) [upload=434656]: Done uploading segments...
Dec 26 06:42:04.788397394 (495h13m4.659480429s) [upload=434656]: Upload finished. err: <nil>
Duration: 30.004316866s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [....................X.......................................]
duration: 
3.559904935s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:42:07 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.559928341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.....................X......................................]
duration: 
5.000832396s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:41:14 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000857674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.................X..........X.......X.X.......X.X.XXX.X.....]
duration: 
5.878586802s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:41:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 5.878615857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [..................X..........X.......X.X.......X.X.XXX.X....]
duration: 
5.900086742s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:40:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 5.900116949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......................X.....................................]
duration: 
3.532678693s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:40:04 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.532709632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [...................X..........X.......X.X.......X.X.XXX.X...]
duration: 
27.44114084s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:39:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 27.441179994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......................X....................................]
duration: 
3.467179119s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:39:27 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.467215948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [....................X..........X.......X.X.......X.X.XXX.X..]
duration: 
30.001992278s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:38:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 30.002018326s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........................X...................................]
duration: 
3.547376099s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:38:24 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.547437395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.....................X..........X.......X.X.......X.X.XXX.X.]
duration: 
5.77791783s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:37:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 5.777942918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................X..................................]
duration: 
3.414196792s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:37:10 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.414223614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [......................X..........X.......X.X.......X.X.XXX.X]
duration: 
30.002690765s
trace: 
https://jaeger.dev.storj.io/trace/610a5a278586d69b
FAIL (2025-12-26 06:36:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded
======LOGS======
Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload
Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload
Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments...
Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false
Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment...
Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment.
Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece...
Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805     , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000
Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment
Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments...
Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482   , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446   , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114    , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942   , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617   , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226    , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976     , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842     , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940     , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189    , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153    , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010     , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193   , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil>
Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264     , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762   , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold
Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments...
Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil>
Duration: 30.002713298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................X.................................]
duration: 
3.898275652s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:35:56 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.898303334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [X...........................................................]
duration: 
25.641121815s
trace: 
https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2
FAIL (2025-12-26 06:35:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded
======LOGS======
upload=433902]: Uploading piece...
Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment
Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments...
Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893     , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033     , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622     , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355   , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212   , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675     , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces.
Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection
Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection
Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection
Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection
Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments...
Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil>
Duration: 25.641153345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [.......................X..........X.......X.X.......X.X.XXX.]
duration: 
5.978925355s
trace: 
https://jaeger.dev.storj.io/trace/6a0a037a6e3db5ee
FAIL (2025-12-26 06:35:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA) context deadline exceeded
======LOGS======
Dec 26 06:35:04.760439678 (495h6m4.631522723s) [upload=434555]: Starting upload
Dec 26 06:35:04.760570547 (495h6m4.631653592s) [upload=434555]: Done starting upload
Dec 26 06:35:04.761041794 (495h6m4.63212484s) [upload=434555]: Uploading segments...
Dec 26 06:35:04.761144149 (495h6m4.632227225s) [upload=434555]: Got next segment. Inline: false
Dec 26 06:35:04.761199875 (495h6m4.63228292s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:35:04.761208602 (495h6m4.632291647s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:35:05.021427045 (495h6m4.89251009s) [seg_pos={0 0}, upload=434555]: Begin upload segment...
Dec 26 06:35:05.021491608 (495h6m4.892574643s) [seg_pos={0 0}, upload=434555]: Done begin upload segment.
Dec 26 06:35:05.021597740 (495h6m4.892680785s) [noise=true , address=45.91.116.10:4084    , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.021610594 (495h6m4.892693639s) [noise=true , address=145.79.227.10:4403   , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.022299917 (495h6m4.893382962s) [noise=true , address=45.91.116.7:4010     , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.022358308 (495h6m4.893441353s) [noise=true , address=145.79.227.33:4870   , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.023752794 (495h6m4.894835839s) [noise=true , address=45.91.116.12:4228    , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024211778 (495h6m4.895294823s) [noise=true , address=77.90.31.13:4627     , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024352175 (495h6m4.89543522s) [noise=true , address=77.90.31.19:4844     , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024663718 (495h6m4.895746763s) [noise=true , address=45.91.116.13:4156    , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.025025226 (495h6m4.896108271s) [noise=true , address=145.79.227.30:4762   , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.025622353 (495h6m4.896705398s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.026734841 (495h6m4.897817886s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.027852760 (495h6m4.898935815s) [noise=true , address=145.79.227.2:4114    , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.028906527 (495h6m4.899989582s) [noise=true , address=145.79.227.16:4911   , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.029388084 (495h6m4.900471119s) [noise=true , address=45.91.116.4:4940     , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.029900950 (495h6m4.900983995s) [noise=true , address=77.90.31.12:4589     , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.031513151 (495h6m4.902596196s) [noise=true , address=45.91.116.14:4191    , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:06.451450055 (495h6m6.32253313s) [noise=true , address=77.90.31.12:4589     , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 26 06:35:06.745433702 (495h6m6.616516737s) [upload=434555]: Next returned nil segment
Dec 26 06:35:06.745449311 (495h6m6.616532346s) [upload=434555]: Waiting for error group managing segments...
Dec 26 06:35:07.033287466 (495h6m6.904370511s) [noise=true , address=145.79.227.2:4114    , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.035687747 (495h6m6.906770792s) [noise=true , address=145.79.227.16:4911   , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.039272383 (495h6m6.910355428s) [noise=true , address=145.79.227.10:4403   , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.059443593 (495h6m6.930526638s) [noise=true , address=145.79.227.33:4870   , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.226574209 (495h6m7.097657254s) [noise=true , address=77.90.31.19:4844     , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.267972800 (495h6m7.139055845s) [noise=true , address=45.91.116.14:4191    , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.270181756 (495h6m7.141264801s) [noise=true , address=45.91.116.7:4010     , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.295349880 (495h6m7.166432925s) [noise=true , address=45.91.116.4:4940     , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.296061265 (495h6m7.1671443s) [noise=true , address=77.90.31.13:4627     , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.304767473 (495h6m7.175850508s) [noise=true , address=45.91.116.13:4156    , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.309903673 (495h6m7.180986718s) [noise=true , address=45.91.116.12:4228    , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.710009747 (495h6m7.581092792s) [noise=true , address=45.91.116.10:4084    , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:08.344914229 (495h6m8.215997264s) [noise=true , address=145.79.227.30:4762   , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:08.344953844 (495h6m8.21603689s) [seg_pos={0 0}, upload=434555]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:35:08.345232144 (495h6m8.216315199s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 26 06:35:08.345243626 (495h6m8.216326671s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold
Dec 26 06:35:08.345263194 (495h6m8.216346239s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 26 06:35:08.345286137 (495h6m8.216369172s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold
Dec 26 06:35:08.345304432 (495h6m8.216387477s) [seg_pos={0 0}, upload=434555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:35:08.345402459 (495h6m8.216485494s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:35:08.345407779 (495h6m8.216490824s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:35:08.636899841 (495h6m8.507982896s) [upload=434555]: Done uploading segments...
Dec 26 06:35:08.636924358 (495h6m8.508007403s) [upload=434555]: Upload finished. err: <nil>
Duration: 5.978956814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [X.......................X..........X.......X.X.......X.X.XXX]
duration: 
30.002575512s
trace: 
https://jaeger.dev.storj.io/trace/6a0a037a6e3db5ee
FAIL (2025-12-26 06:35:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA) context deadline exceeded
======LOGS======
Dec 26 06:35:04.760439678 (495h6m4.631522723s) [upload=434555]: Starting upload
Dec 26 06:35:04.760570547 (495h6m4.631653592s) [upload=434555]: Done starting upload
Dec 26 06:35:04.761041794 (495h6m4.63212484s) [upload=434555]: Uploading segments...
Dec 26 06:35:04.761144149 (495h6m4.632227225s) [upload=434555]: Got next segment. Inline: false
Dec 26 06:35:04.761199875 (495h6m4.63228292s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:35:04.761208602 (495h6m4.632291647s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:35:05.021427045 (495h6m4.89251009s) [seg_pos={0 0}, upload=434555]: Begin upload segment...
Dec 26 06:35:05.021491608 (495h6m4.892574643s) [seg_pos={0 0}, upload=434555]: Done begin upload segment.
Dec 26 06:35:05.021597740 (495h6m4.892680785s) [noise=true , address=45.91.116.10:4084    , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.021610594 (495h6m4.892693639s) [noise=true , address=145.79.227.10:4403   , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.022299917 (495h6m4.893382962s) [noise=true , address=45.91.116.7:4010     , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.022358308 (495h6m4.893441353s) [noise=true , address=145.79.227.33:4870   , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.023752794 (495h6m4.894835839s) [noise=true , address=45.91.116.12:4228    , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024211778 (495h6m4.895294823s) [noise=true , address=77.90.31.13:4627     , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024352175 (495h6m4.89543522s) [noise=true , address=77.90.31.19:4844     , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.024663718 (495h6m4.895746763s) [noise=true , address=45.91.116.13:4156    , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.025025226 (495h6m4.896108271s) [noise=true , address=145.79.227.30:4762   , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.025622353 (495h6m4.896705398s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.026734841 (495h6m4.897817886s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.027852760 (495h6m4.898935815s) [noise=true , address=145.79.227.2:4114    , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.028906527 (495h6m4.899989582s) [noise=true , address=145.79.227.16:4911   , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.029388084 (495h6m4.900471119s) [noise=true , address=45.91.116.4:4940     , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.029900950 (495h6m4.900983995s) [noise=true , address=77.90.31.12:4589     , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:05.031513151 (495h6m4.902596196s) [noise=true , address=45.91.116.14:4191    , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Uploading piece...
Dec 26 06:35:06.451450055 (495h6m6.32253313s) [noise=true , address=77.90.31.12:4589     , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 26 06:35:06.745433702 (495h6m6.616516737s) [upload=434555]: Next returned nil segment
Dec 26 06:35:06.745449311 (495h6m6.616532346s) [upload=434555]: Waiting for error group managing segments...
Dec 26 06:35:07.033287466 (495h6m6.904370511s) [noise=true , address=145.79.227.2:4114    , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.035687747 (495h6m6.906770792s) [noise=true , address=145.79.227.16:4911   , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.039272383 (495h6m6.910355428s) [noise=true , address=145.79.227.10:4403   , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.059443593 (495h6m6.930526638s) [noise=true , address=145.79.227.33:4870   , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.226574209 (495h6m7.097657254s) [noise=true , address=77.90.31.19:4844     , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.267972800 (495h6m7.139055845s) [noise=true , address=45.91.116.14:4191    , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.270181756 (495h6m7.141264801s) [noise=true , address=45.91.116.7:4010     , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.295349880 (495h6m7.166432925s) [noise=true , address=45.91.116.4:4940     , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.296061265 (495h6m7.1671443s) [noise=true , address=77.90.31.13:4627     , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.304767473 (495h6m7.175850508s) [noise=true , address=45.91.116.13:4156    , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.309903673 (495h6m7.180986718s) [noise=true , address=45.91.116.12:4228    , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:07.710009747 (495h6m7.581092792s) [noise=true , address=45.91.116.10:4084    , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:08.344914229 (495h6m8.215997264s) [noise=true , address=145.79.227.30:4762   , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil>
Dec 26 06:35:08.344953844 (495h6m8.21603689s) [seg_pos={0 0}, upload=434555]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:35:08.345232144 (495h6m8.216315199s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 26 06:35:08.345243626 (495h6m8.216326671s) [noise=true , address=45.91.116.11:4118    , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold
Dec 26 06:35:08.345263194 (495h6m8.216346239s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 26 06:35:08.345286137 (495h6m8.216369172s) [noise=true , address=145.79.227.44:4195   , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold
Dec 26 06:35:08.345304432 (495h6m8.216387477s) [seg_pos={0 0}, upload=434555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:35:08.345402459 (495h6m8.216485494s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:35:08.345407779 (495h6m8.216490824s) [upload=434555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:35:08.636899841 (495h6m8.507982896s) [upload=434555]: Done uploading segments...
Dec 26 06:35:08.636924358 (495h6m8.508007403s) [upload=434555]: Upload finished. err: <nil>
Duration: 30.002630085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration: 
30.595842882s
trace: 
https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2
FAIL (2025-12-26 06:35:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded
======LOGS======
upload=433902]: Uploading piece...
Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment
Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments...
Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893     , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033     , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622     , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355   , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212   , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675     , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces.
Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection
Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection
Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection
Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection
Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments...
Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil>
Duration: 30.59587918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................X................................]
duration: 
3.455899903s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:34:43 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.455923458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration: 
30.783074238s
trace: 
https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2
FAIL (2025-12-26 06:34:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded
======LOGS======
upload=433902]: Uploading piece...
Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece...
Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment
Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments...
Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509    , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579     , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204    , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755     , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653     , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893     , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033     , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622     , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355   , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522     , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936   , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212   , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675     , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890   , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil>
Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces.
Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection
Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381   , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection
Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785   , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection
Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052   , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection
Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461   , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold
Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments...
Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil>
Duration: 30.783098083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [.X.......................X..........X.......X.X.......X.X.XX]
duration: 
30.002132134s
trace: 
https://jaeger.dev.storj.io/trace/55c6e26a832cd8a1
FAIL (2025-12-26 06:34:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ) context deadline exceeded
======LOGS======
Dec 26 06:33:53.016580725 (495h4m52.88766376s) [upload=434536]: Starting upload
Dec 26 06:33:53.016645288 (495h4m52.887728323s) [upload=434536]: Done starting upload
Dec 26 06:33:53.016878282 (495h4m52.887961327s) [upload=434536]: Uploading segments...
Dec 26 06:33:53.017048286 (495h4m52.888131331s) [upload=434536]: Got next segment. Inline: false
Dec 26 06:33:53.017107208 (495h4m52.888190243s) [upload=434536]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:33:53.017115543 (495h4m52.888198578s) [upload=434536]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:33:53.263867924 (495h4m53.134950969s) [seg_pos={0 0}, upload=434536]: Begin upload segment...
Dec 26 06:33:53.263920705 (495h4m53.13500375s) [seg_pos={0 0}, upload=434536]: Done begin upload segment.
Dec 26 06:33:53.263950472 (495h4m53.135033517s) [noise=true , address=145.79.227.37:4565   , piece_id=YOFIF6SKF7JYWCYKYKQ2AMGD6PG6D7ZGKXEMAZPYMJQZSHB7TSKQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.264900060 (495h4m53.135983105s) [noise=true , address=45.91.116.4:4939     , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.265321652 (495h4m53.136404697s) [noise=true , address=145.79.227.14:4837   , piece_id=BE5WQ2KEA5NQWKSXZCC4HZVJYFHDZLXF62AJGIY2TTRNKIVJJT2A, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.265727586 (495h4m53.136810631s) [noise=true , address=145.79.227.49:4337   , piece_id=MBOL7EEJBFFVFNUZEYNCZZCKOD6PPMOYLIK6LGBOBYLKCTII77FQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.266143929 (495h4m53.137226964s) [noise=true , address=45.91.116.6:4974     , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.266630715 (495h4m53.137713761s) [noise=true , address=45.91.116.7:4011     , piece_id=7HRWVQJADD5DVU73NLOQYN6ROFZSXPZO3Y532GICQEIIK5B3ISQA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.267087495 (495h4m53.13817054s) [noise=true , address=45.91.116.5:4299     , piece_id=ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.267485302 (495h4m53.138568357s) [noise=true , address=145.79.227.41:4085   , piece_id=CHR44GT7FQEV2GCVGKNOJA5XCRHRAPHACLT74BWXVLMAUK3URZDQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.267858623 (495h4m53.138941668s) [noise=true , address=45.91.116.14:4189    , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.268257322 (495h4m53.139340367s) [noise=true , address=145.79.227.32:4835   , piece_id=HY6SZRVYSCAE3KU2SIRJYQHNF66DM6QNTWIVLTBAIQHEIXOKEYRA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.268636644 (495h4m53.13971969s) [noise=true , address=145.79.227.47:4266   , piece_id=5CKGHLDR6IFFUKSPGAOEWBEMQSKWSXO26MT2QLNRZETL4UKBTOYA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.269091911 (495h4m53.140174956s) [noise=true , address=45.91.116.9:4262     , piece_id=WMKK6KZ5XQZF3SUMWXIG4WOYV765G23JHHM2LLX2ACK7RRMGJDLQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.269563550 (495h4m53.140646605s) [noise=true , address=145.79.227.34:4907   , piece_id=TTSL3SIME3JX3ABNMGSIAI7RFYHXBSPX7GE5AKMUTRXR5C3BXZXQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.270267911 (495h4m53.141350966s) [noise=true , address=45.91.116.11:4117    , piece_id=E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.270821675 (495h4m53.14190471s) [noise=true , address=145.79.227.17:4970   , piece_id=NNATAFYU3UT4JOORYI4V5Y65FBHIAE4LAN6U65VYDKXVFSRA2REQ, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:53.271252806 (495h4m53.142335841s) [noise=true , address=145.79.227.10:4403   , piece_id=AYJ2DYFWG7FIMXA45G37MNVTR24MS3AIJ5KALYOY6FV5YSMNLTVA, seg_pos={0 0}, upload=434536]: Uploading piece...
Dec 26 06:33:54.990958485 (495h4m54.86204152s) [upload=434536]: Next returned nil segment
Dec 26 06:33:54.990980767 (495h4m54.862063802s) [upload=434536]: Waiting for error group managing segments...
Dec 26 06:33:55.278685418 (495h4m55.149768463s) [noise=true , address=145.79.227.14:4837   , piece_id=BE5WQ2KEA5NQWKSXZCC4HZVJYFHDZLXF62AJGIY2TTRNKIVJJT2A, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.278727087 (495h4m55.149810122s) [noise=true , address=145.79.227.41:4085   , piece_id=CHR44GT7FQEV2GCVGKNOJA5XCRHRAPHACLT74BWXVLMAUK3URZDQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.280052541 (495h4m55.151135586s) [noise=true , address=145.79.227.37:4565   , piece_id=YOFIF6SKF7JYWCYKYKQ2AMGD6PG6D7ZGKXEMAZPYMJQZSHB7TSKQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.280159625 (495h4m55.15124267s) [noise=true , address=145.79.227.17:4970   , piece_id=NNATAFYU3UT4JOORYI4V5Y65FBHIAE4LAN6U65VYDKXVFSRA2REQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.286058809 (495h4m55.157141854s) [noise=true , address=145.79.227.10:4403   , piece_id=AYJ2DYFWG7FIMXA45G37MNVTR24MS3AIJ5KALYOY6FV5YSMNLTVA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.286307522 (495h4m55.157390567s) [noise=true , address=145.79.227.49:4337   , piece_id=MBOL7EEJBFFVFNUZEYNCZZCKOD6PPMOYLIK6LGBOBYLKCTII77FQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.286848011 (495h4m55.157931046s) [noise=true , address=145.79.227.32:4835   , piece_id=HY6SZRVYSCAE3KU2SIRJYQHNF66DM6QNTWIVLTBAIQHEIXOKEYRA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.293410307 (495h4m55.164493362s) [noise=true , address=145.79.227.47:4266   , piece_id=5CKGHLDR6IFFUKSPGAOEWBEMQSKWSXO26MT2QLNRZETL4UKBTOYA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.305200408 (495h4m55.176283443s) [noise=true , address=145.79.227.34:4907   , piece_id=TTSL3SIME3JX3ABNMGSIAI7RFYHXBSPX7GE5AKMUTRXR5C3BXZXQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.480806763 (495h4m55.351889808s) [noise=true , address=45.91.116.9:4262     , piece_id=WMKK6KZ5XQZF3SUMWXIG4WOYV765G23JHHM2LLX2ACK7RRMGJDLQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.543437063 (495h4m55.414520098s) [noise=true , address=45.91.116.11:4117    , piece_id=E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.545101342 (495h4m55.416184397s) [noise=true , address=45.91.116.5:4299     , piece_id=ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.632390887 (495h4m55.503473922s) [noise=true , address=45.91.116.7:4011     , piece_id=7HRWVQJADD5DVU73NLOQYN6ROFZSXPZO3Y532GICQEIIK5B3ISQA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil>
Dec 26 06:33:55.632407980 (495h4m55.503491025s) [seg_pos={0 0}, upload=434536]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:33:55.632638809 (495h4m55.503721854s) [noise=true , address=45.91.116.6:4974     , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:55.632648167 (495h4m55.503731222s) [noise=true , address=45.91.116.6:4974     , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold
Dec 26 06:33:55.632951524 (495h4m55.504034569s) [noise=true , address=45.91.116.14:4189    , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:55.632960382 (495h4m55.504043447s) [noise=true , address=45.91.116.14:4189    , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold
Dec 26 06:33:55.633131136 (495h4m55.504214191s) [noise=true , address=45.91.116.4:4939     , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:55.633139322 (495h4m55.504222387s) [noise=true , address=45.91.116.4:4939     , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold
Dec 26 06:33:55.633158568 (495h4m55.504241623s) [seg_pos={0 0}, upload=434536]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:33:55.633243360 (495h4m55.504326415s) [upload=434536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:33:55.633250073 (495h4m55.504333118s) [upload=434536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:33:55.930018343 (495h4m55.801101378s) [upload=434536]: Done uploading segments...
Dec 26 06:33:55.930041076 (495h4m55.801124121s) [upload=434536]: Upload finished. err: <nil>
Duration: 30.002145319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [X.X.......................X..........X.......X.X.......X.X.X]
duration: 
30.004187788s
trace: 
https://jaeger.dev.storj.io/trace/1de491d2df40f807
FAIL (2025-12-26 06:33:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ) context deadline exceeded
======LOGS======
Dec 26 06:33:22.768741754 (495h4m22.639824809s) [upload=434528]: Starting upload
Dec 26 06:33:22.768854699 (495h4m22.639937734s) [upload=434528]: Done starting upload
Dec 26 06:33:22.768943237 (495h4m22.640026282s) [upload=434528]: Uploading segments...
Dec 26 06:33:22.769259761 (495h4m22.640342836s) [upload=434528]: Got next segment. Inline: false
Dec 26 06:33:22.769341736 (495h4m22.640424781s) [upload=434528]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 06:33:22.769350272 (495h4m22.640433327s) [upload=434528]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 06:33:23.016278597 (495h4m22.887361662s) [seg_pos={0 0}, upload=434528]: Begin upload segment...
Dec 26 06:33:23.016337560 (495h4m22.887420605s) [seg_pos={0 0}, upload=434528]: Done begin upload segment.
Dec 26 06:33:23.016370502 (495h4m22.887453537s) [noise=true , address=145.79.227.2:4114    , piece_id=5QI6IKKKEXD5O4QNMGEXODLTVUT75U6MFMUWCLV5LWWU3P6SCKJA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.016499377 (495h4m22.887582422s) [noise=true , address=77.90.31.11:4556     , piece_id=D2SC3D5KONTBNLL5SSFX3LOSJ36PYQCMITOOBQ4NLGA6E444GKGA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.017152952 (495h4m22.888235997s) [noise=true , address=45.91.116.12:4225    , piece_id=OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.017186676 (495h4m22.888269721s) [noise=true , address=45.91.116.11:4120    , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.017585366 (495h4m22.888668411s) [noise=true , address=145.79.227.33:4869   , piece_id=K47XWKUNBZNHGMC65DCHXUH3G7AJUEAD7O2ENIYPYXW7LZTBC6LA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.017614581 (495h4m22.888697626s) [noise=true , address=145.79.227.19:4079   , piece_id=EJPYP57RF4RKMIWNGE3Q3N3ULC7BWYFV245OYDUK36W2OXWNTY5Q, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.018040883 (495h4m22.889123928s) [noise=true , address=45.91.116.6:4973     , piece_id=NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.018086269 (495h4m22.889169304s) [noise=true , address=145.79.227.45:4231   , piece_id=2WMB4UBCMOGDZCB6NIIS644QUHDL27SYXKT4O37ZWL3UCZHANYFA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.018532980 (495h4m22.889616025s) [noise=true , address=145.79.227.51:4410   , piece_id=TX3DMZVKZKQQDKCRLSRXE3DDJ6EKKGPC7RGOLKAW3CLHVK2BKI4Q, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.018721128 (495h4m22.889804173s) [noise=true , address=145.79.227.6:4259    , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.018939213 (495h4m22.890022258s) [noise=true , address=45.91.116.13:4155    , piece_id=GKTHXIZQM3CSIBNLI5CYBOAZUW25BFD2HTVB23FG3QTHDUTIZWOQ, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.019205420 (495h4m22.890288455s) [noise=true , address=45.91.116.14:4190    , piece_id=NC324SPWATSYMJO2GQIUSYVIPMLL55WZJSGT6QKNGMOEQNQT67PA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.019357399 (495h4m22.890440444s) [noise=true , address=45.91.116.7:4011     , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.019606204 (495h4m22.890689249s) [noise=true , address=77.90.31.14:4663     , piece_id=BSA54IMJOEW5GHMBTJD55BVZBHVGOQRBOTIKCMKMZQWHQFB2HHMA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.019750108 (495h4m22.890833143s) [noise=true , address=45.91.116.8:4045     , piece_id=S754DS2BZI5USNAZHN4LM6DAV6PUGQN5B76D2UGUFHSVB33U2NEQ, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:23.020060118 (495h4m22.891143163s) [noise=true , address=145.79.227.39:4013   , piece_id=4ILC2PH5YARWONAFXY3PQC2IMFJGUHGQVFMQSXW2U5L7UYKHEPTA, seg_pos={0 0}, upload=434528]: Uploading piece...
Dec 26 06:33:24.739655254 (495h4m24.610738289s) [upload=434528]: Next returned nil segment
Dec 26 06:33:24.739673548 (495h4m24.610756593s) [upload=434528]: Waiting for error group managing segments...
Dec 26 06:33:25.026396559 (495h4m24.897479594s) [noise=true , address=145.79.227.51:4410   , piece_id=TX3DMZVKZKQQDKCRLSRXE3DDJ6EKKGPC7RGOLKAW3CLHVK2BKI4Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.028254748 (495h4m24.899337793s) [noise=true , address=145.79.227.2:4114    , piece_id=5QI6IKKKEXD5O4QNMGEXODLTVUT75U6MFMUWCLV5LWWU3P6SCKJA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.029641289 (495h4m24.900724334s) [noise=true , address=145.79.227.19:4079   , piece_id=EJPYP57RF4RKMIWNGE3Q3N3ULC7BWYFV245OYDUK36W2OXWNTY5Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.030902690 (495h4m24.901985735s) [noise=true , address=145.79.227.39:4013   , piece_id=4ILC2PH5YARWONAFXY3PQC2IMFJGUHGQVFMQSXW2U5L7UYKHEPTA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.048536449 (495h4m24.919619494s) [noise=true , address=145.79.227.45:4231   , piece_id=2WMB4UBCMOGDZCB6NIIS644QUHDL27SYXKT4O37ZWL3UCZHANYFA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.058889051 (495h4m24.929972096s) [noise=true , address=145.79.227.33:4869   , piece_id=K47XWKUNBZNHGMC65DCHXUH3G7AJUEAD7O2ENIYPYXW7LZTBC6LA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.234118720 (495h4m25.105201755s) [noise=true , address=77.90.31.11:4556     , piece_id=D2SC3D5KONTBNLL5SSFX3LOSJ36PYQCMITOOBQ4NLGA6E444GKGA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.309219606 (495h4m25.180302641s) [noise=true , address=45.91.116.13:4155    , piece_id=GKTHXIZQM3CSIBNLI5CYBOAZUW25BFD2HTVB23FG3QTHDUTIZWOQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.323837601 (495h4m25.194920636s) [noise=true , address=77.90.31.14:4663     , piece_id=BSA54IMJOEW5GHMBTJD55BVZBHVGOQRBOTIKCMKMZQWHQFB2HHMA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.370023107 (495h4m25.241106152s) [noise=true , address=45.91.116.12:4225    , piece_id=OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.395370583 (495h4m25.266453648s) [noise=true , address=45.91.116.8:4045     , piece_id=S754DS2BZI5USNAZHN4LM6DAV6PUGQN5B76D2UGUFHSVB33U2NEQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.893927055 (495h4m25.76501009s) [noise=true , address=45.91.116.14:4190    , piece_id=NC324SPWATSYMJO2GQIUSYVIPMLL55WZJSGT6QKNGMOEQNQT67PA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.931758572 (495h4m25.802841607s) [noise=true , address=45.91.116.6:4973     , piece_id=NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil>
Dec 26 06:33:25.931780895 (495h4m25.80286394s) [seg_pos={0 0}, upload=434528]: Segment reached optimal threshold of 13 pieces.
Dec 26 06:33:25.931995663 (495h4m25.803078709s) [noise=true , address=145.79.227.6:4259    , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:25.932004942 (495h4m25.803087987s) [noise=true , address=145.79.227.6:4259    , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold
Dec 26 06:33:25.932111956 (495h4m25.803194991s) [noise=true , address=45.91.116.7:4011     , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:25.932121614 (495h4m25.803204659s) [noise=true , address=45.91.116.7:4011     , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold
Dec 26 06:33:25.932259306 (495h4m25.803342351s) [noise=true , address=45.91.116.11:4120    , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 06:33:25.932266800 (495h4m25.803349845s) [noise=true , address=45.91.116.11:4120    , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold
Dec 26 06:33:25.932285246 (495h4m25.803368291s) [seg_pos={0 0}, upload=434528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 06:33:25.932361330 (495h4m25.803444375s) [upload=434528]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 06:33:25.932368544 (495h4m25.803451579s) [upload=434528]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 06:33:26.241366512 (495h4m26.112449547s) [upload=434528]: Done uploading segments...
Dec 26 06:33:26.241388052 (495h4m26.112471097s) [upload=434528]: Upload finished. err: <nil>
Duration: 30.004213617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............................X...............................]
duration: 
3.44897796s
trace: 
https://jaeger.dev.storj.io/trace/630094597d401477
FAIL (2025-12-26 06:33:43 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.44900888s core uplink us-select-1 us1-placement-20