Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.959244309s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:04:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.959265738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.384310687s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:04:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.384371693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 5.864692649s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:04:04 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.864723067s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 5.001380668s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:03:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001409723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX...............................X....X.................] duration: 5.773784149s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.773807244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.961040818s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:03:26 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.96107347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....XX...............................X....X................] duration: 5.700736271s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:03:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.700779924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 5.780651104s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:03:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.780684979s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....XX...............................X....X...............] duration: 5.856676888s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856721703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.067392765s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:02:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.067423002s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.555368324s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:02:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555396267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....XX...............................X....X..............] duration: 5.797134865s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:02:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.797166245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......X..............................................] duration: 5.80398891s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:02:17 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.804013677s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....XX...............................X....X.............] duration: 5.802218682s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:01:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.802248079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.036504847s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:01:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.036535327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X....XX...............................X....X............] duration: 5.768819793s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:01:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.768848709s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.407136992s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:01:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407170066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration: 5.706102771s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:01:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.706125435s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X....XX...............................X....X...........] duration: 5.774600495s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:01:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.774631734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.004638536s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:01:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.004662823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X......X............................................] duration: 5.955098779s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 16:00:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.955128265s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.999517186s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:00:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.999561269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.988789638s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 16:00:30 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.988866985s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X....XX...............................X....X..........] duration: 6.248129768s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 16:00:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.24816823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.982813587s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 16:00:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 1.982874442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.529312902s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 16:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529362485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X....XX...............................X....X.........] duration: 5.701212042s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:59:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.701249183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X......X...........................................] duration: 5.86794196s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:59:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.867972318s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.046709259s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:59:55 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.046729918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.95997498s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:59:50 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.960045584s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X....XX...............................X....X........] duration: 5.781088209s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:59:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.781120429s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.622316627s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:59:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.622424534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X......X..........................................] duration: 5.718169078s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:58:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.718200359s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.04109615s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:58:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.041125086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.968953113s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:58:40 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.968978533s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X....XX...............................X....X.......] duration: 5.756590591s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.756622812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X......X.........................................] duration: 5.546972715s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:58:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.546998904s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.653886877s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:57:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.653916894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.083682545s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:57:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.083731116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X......X........................................] duration: 5.78465347s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:57:47 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.784738182s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.971645472s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:57:38 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.971676401s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X....XX...............................X....X......] duration: 6.660860291s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.660902393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.051925059s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:57:07 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.051956749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X....XX...............................X....X.....] duration: 5.861507445s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:56:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.8615371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X......X.......................................] duration: 5.906530106s trace: https://jaeger.dev.storj.io/trace/2c2786b5f1ef2e7 FAIL (2025-11-29 15:56:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.906555705s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.966285665s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:56:44 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.966357923s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.437229132s trace: https://jaeger.dev.storj.io/trace/6f5a1c3cd166f13e FAIL (2025-11-29 15:56:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437262355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.074271932s trace: https://jaeger.dev.storj.io/trace/3e63ccb8995d0c74 FAIL (2025-11-29 15:56:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 29 15:30:21.077010022 (487h26m19.628480483s) [upload=429217]: Starting upload Nov 29 15:30:21.077080997 (487h26m19.628551458s) [upload=429217]: Done starting upload Nov 29 15:30:21.077162331 (487h26m19.628632793s) [upload=429217]: Uploading segments... Nov 29 15:30:21.077217136 (487h26m19.628687597s) [upload=429217]: Got next segment. Inline: true Nov 29 15:30:21.077222937 (487h26m19.628693388s) [upload=429217]: Waiting for error group managing segments... Nov 29 15:30:21.077272582 (487h26m19.628743063s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:30:21.077279405 (487h26m19.628749866s) [upload=429217]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 29 15:30:21.077283212 (487h26m19.628753673s) [upload=429217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 29 15:30:21.827579183 (487h26m20.379049654s) [upload=429217]: Done uploading segments... Nov 29 15:30:21.827605372 (487h26m20.379075833s) [upload=429217]: Upload finished. err: <nil> Duration:* 2.074299624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X....XX...............................X....X....] duration: 5.838933196s trace: https://jaeger.dev.storj.io/trace/2886e49b2855e9a8 FAIL (2025-11-29 15:56:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== v 29 15:54:17.204005683 (487h50m15.755476144s) [upload=429575]: Got next segment. Inline: false Nov 29 15:54:17.204045460 (487h50m15.755515921s) [upload=429575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 15:54:17.204052122 (487h50m15.755522583s) [upload=429575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 15:54:17.454343340 (487h50m16.005813801s) [seg_pos={0 0}, upload=429575]: Begin upload segment... Nov 29 15:54:17.454402964 (487h50m16.005873425s) [seg_pos={0 0}, upload=429575]: Done begin upload segment. Nov 29 15:54:17.454437470 (487h50m16.005907931s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455426152 (487h50m16.006896613s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.455851982 (487h50m16.007322443s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456341414 (487h50m16.007811875s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.456759701 (487h50m16.008230162s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457170754 (487h50m16.008641215s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.457576396 (487h50m16.009046858s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458012506 (487h50m16.009482967s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458419902 (487h50m16.009890363s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.458835704 (487h50m16.010306165s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459351356 (487h50m16.010821817s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.459765144 (487h50m16.011235605s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460180766 (487h50m16.011651227s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.460582130 (487h50m16.012052591s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461002239 (487h50m16.0124727s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:17.461418111 (487h50m16.012888572s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Uploading piece... Nov 29 15:54:19.148732036 (487h50m17.700202497s) [upload=429575]: Next returned nil segment Nov 29 15:54:19.148752194 (487h50m17.700222655s) [upload=429575]: Waiting for error group managing segments... Nov 29 15:54:19.552449780 (487h50m18.103920241s) [noise=true , address=45.91.116.5:4299 , piece_id=CS5FGLTHZC67TH6RXXAHMM5IQF22NTII65ZGMUCJUZTB5LZMTORQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.556249737 (487h50m18.107720198s) [noise=true , address=45.91.116.13:4153 , piece_id=ZOWHI2CFYG7SRBT6IQHKPHUTDX5POAFWUPIU37NE4NDQ7C75A4LA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:19.581670122 (487h50m18.133140583s) [noise=true , address=77.90.31.17:4771 , piece_id=FKRNNJORPETYPTYHJFR6M43NEX45EYQXZRSY7SAJMMEBHHXACCBA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.249335548 (487h50m18.800806009s) [noise=true , address=145.79.227.21:4475 , piece_id=EVIIL4S6FDUUL2AQXGVDARRXPDBRZ3R4MSYKWS4LAZQALBH5GA4Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.370764655 (487h50m18.922235106s) [noise=true , address=145.79.227.22:4511 , piece_id=AKDWYVEWLP7N4XGQFH5AV67W25FPFGYT7OZHUAQTZQBP7MB7UCPQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.448196182 (487h50m18.999666643s) [noise=true , address=145.79.227.3:4150 , piece_id=BFQ23SR72RHCV2TGZHLJSUX5FFIPAJNEODNKOHTDL2Q6YZ2VRKYQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.571676143 (487h50m19.123146604s) [noise=true , address=145.79.227.27:4689 , piece_id=4XO4TL2PUZNANH3YFJWXNYEYNU7ZK7UNPA67STJHI5RYYLEAHXNA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.626809468 (487h50m19.178279929s) [noise=true , address=145.79.227.8:4330 , piece_id=PPWK5JTBNDJMCIBU3FUK6RVLQDZVTDJ42TCUSTON4JTUDUOVUEZA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.720603693 (487h50m19.272074154s) [noise=true , address=145.79.227.32:4835 , piece_id=FOWJL7Q55QRQOO6L3566OKVMERSZVTLPJ2Q7DNG64QYXKBDULIWQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:20.727830004 (487h50m19.279300465s) [noise=true , address=145.79.227.5:4222 , piece_id=OL54JUX5MDKBRGXZOMMTFMZQY2YKCH3F22TKUQKJM7Z5ATGM4CFQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:21.902515965 (487h50m20.453986426s) [noise=true , address=145.79.227.29:4943 , piece_id=LQYIACU6SG3SL34ZAD624YKUEBOLA7VPDSZ4VL3OROHXOFWFU7GQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: <nil> Nov 29 15:54:46.500764249 (487h50m45.05223472s) [noise=true , address=145.79.227.28:4726 , piece_id=V3Q6XP4OZRIREPGC65D37ULKFAPUCJY4YXNUFT7QXT4KOIAWRBXA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58808->145.79.227.28:4726: use of closed network connection Nov 29 15:54:46.501091543 (487h50m45.052562004s) [noise=true , address=145.79.227.15:4874 , piece_id=HBV6FOE5LCRMFEF5RLO6FX5CB2SGSTPKNW4EVVFXMMBRALKJKJLA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46754->145.79.227.15:4874: use of closed network connection Nov 29 15:54:46.501330637 (487h50m45.052801108s) [noise=true , address=145.79.227.6:4258 , piece_id=DFFHELEE66RK36QSN72DRZKEPNHZHQ4S37WWGALU7CO344AS4S6Q, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40996->145.79.227.6:4258: use of closed network connection Nov 29 15:54:46.501514358 (487h50m45.052984819s) [noise=true , address=145.79.227.7:4294 , piece_id=MSXHHYK3VFGI7KX64MD7WGZTIX6OFGJNRM5O67NG65YVAM7EDTCA, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58922->145.79.227.7:4294: use of closed network connection Nov 29 15:54:46.501645908 (487h50m45.053116369s) [noise=true , address=145.79.227.25:4618 , piece_id=S7I3JOODLSSRDJB3ROEFSPHIIVX2IDDL22LZJDMGOI2OBRBEMPGQ, seg_pos={0 0}, upload=429575]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58804->145.79.227.25:4618: use of closed network connection Nov 29 15:54:46.501700221 (487h50m45.053170682s) [seg_pos={0 0}, upload=429575]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 15:54:46.749528489 (487h50m45.30099895s) [upload=429575]: Done uploading segments... Nov 29 15:54:46.749570438 (487h50m45.3010409s) [upload=429575]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.838972541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.978495997s trace: https://jaeger.dev.storj.io/trace/1e7890c776285ed9 FAIL (2025-11-29 15:56:14 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.97852868s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)