Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.918754525s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:21:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.91878293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.027915314s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.027947816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.803367889s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.803394359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.009865213s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.009903497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.001911892s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.002097985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.936328759s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.936354336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.990056355s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:17:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.990086482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.965253326s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.965299454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.971342571s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.971368641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.847074275s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:16:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.8471081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.83694677s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.837041311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.899483138s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:14:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.899514469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.910504102s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:13:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.910552123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.914369716s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:12:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.914399633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.862723779s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:11:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.862756211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.914177566s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:10:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.914212302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.799148582s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:10:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799258421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.941462158s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:09:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.941495221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.795071148s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.795097238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.936262157s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:07:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.936291895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.881514497s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:07:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.881533805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.247013026s trace: https://jaeger.dev.storj.io/trace/6e6c919d95105c58 FAIL (2025-11-21 04:06:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== :26.528319511 (284h2m25.079789972s) [upload=250173]: Done starting upload Nov 21 04:06:26.528381539 (284h2m25.079852s) [upload=250173]: Uploading segments... Nov 21 04:06:26.528513070 (284h2m25.079983531s) [upload=250173]: Got next segment. Inline: false Nov 21 04:06:26.528554950 (284h2m25.080025411s) [upload=250173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 04:06:26.528560039 (284h2m25.08003049s) [upload=250173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 04:06:26.780874126 (284h2m25.332344588s) [seg_pos={0 0}, upload=250173]: Begin upload segment... Nov 21 04:06:26.780942627 (284h2m25.332413098s) [seg_pos={0 0}, upload=250173]: Done begin upload segment. Nov 21 04:06:26.780980109 (284h2m25.33245058s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781105848 (284h2m25.332576299s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.781682226 (284h2m25.333152687s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782086435 (284h2m25.333556896s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782466890 (284h2m25.333937351s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.782895536 (284h2m25.334365997s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783395358 (284h2m25.334865819s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.783843842 (284h2m25.335314303s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.784321481 (284h2m25.335791942s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.786637451 (284h2m25.338107912s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.788495890 (284h2m25.339966361s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.790346503 (284h2m25.341816964s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.794625201 (284h2m25.346095662s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.797944322 (284h2m25.349414783s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.803903539 (284h2m25.35537401s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:26.805302673 (284h2m25.356773134s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Uploading piece... Nov 21 04:06:28.572999401 (284h2m27.124469863s) [upload=250173]: Next returned nil segment Nov 21 04:06:28.573015602 (284h2m27.124486063s) [upload=250173]: Waiting for error group managing segments... Nov 21 04:06:28.860043331 (284h2m27.411513792s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFALJJDAYOVCCUHMHRJWONBKYK72TA3MAVL6PBMKFUBYSF53AYMA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.886399026 (284h2m27.437869488s) [noise=true , address=45.91.116.6:4973 , piece_id=H53ECBBW6LGIFLGLZLD2E7YHYYEONT6P7MDZDREYAGQVR4CMU44Q, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.888414074 (284h2m27.439884535s) [noise=true , address=45.91.116.4:4940 , piece_id=H5QV5GCVJOCIBDJXDDUEVOQDJUIPQLQKMBBB2OCMGMYQNDLB2JBA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:28.963936220 (284h2m27.515406671s) [noise=true , address=77.90.31.12:4590 , piece_id=7LMAZIELSYB64PFGE6PQ5NY3PKT4P7IGYSTBTFGIH7RBO3S2BWMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:29.900168993 (284h2m28.451639454s) [noise=true , address=145.79.227.6:4257 , piece_id=52CRJXIEJVLR6JJGMNFC54BH3MT2JDBMMDTDSLIDOOC53G756ZCQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.116930223 (284h2m28.668400684s) [noise=true , address=145.79.227.23:4546 , piece_id=4EQTBJ6UKM5LYYS4G7ZPO4WXM2SS4ETY4T7T7M6EHQUOPOUHIMHA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.209523596 (284h2m28.760994057s) [noise=true , address=145.79.227.10:4401 , piece_id=FTRSM2BIAM6ONORXYDXLJEJ2BZHNRK5NPQIJGGPHA77EVEYVKCXA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.304254248 (284h2m28.855724709s) [noise=true , address=145.79.227.15:4873 , piece_id=GQUTOHB3SUAZKX77F5XBGSLTZHZCRTR7QLOQDRRM7CAMMNCT6GTA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:30.800691568 (284h2m29.352162029s) [noise=true , address=145.79.227.22:4509 , piece_id=RLWBNDH26BS742X64VD64DMRH4RIB5JLUY4WYFLM2W7ZU5PJYVDQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:31.110421525 (284h2m29.661891986s) [noise=true , address=145.79.227.30:4761 , piece_id=XI23LSPXSOCS57F3U5HKPPMYZAKHDOI5K5QRYXUTICFDBJ2LZLVQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:33.331936218 (284h2m31.883406679s) [noise=true , address=145.79.227.9:4367 , piece_id=23423YBJDBP2JYW3AV6CKP2QF4BT5FVBCCVSSNDVTDNDOE7KUSMQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: <nil> Nov 21 04:06:55.835800062 (284h2m54.387270523s) [noise=true , address=145.79.227.32:4836 , piece_id=TBQF5F7MGZDQRYJKS5IYLM46YRIQGJOCY57765W3FKAD5ZYOGEPQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50762->145.79.227.32:4836: use of closed network connection Nov 21 04:06:55.835941671 (284h2m54.387412132s) [noise=true , address=145.79.227.11:4731 , piece_id=JPDUWMGCKLYQO3ZHG4GY7MBJE3EMPYMMJ27L24VOUMIUMXDNSXJQ, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51930->145.79.227.11:4731: use of closed network connection Nov 21 04:06:55.836058544 (284h2m54.387529005s) [noise=true , address=145.79.227.8:4330 , piece_id=ZQILMHKQIGWMBSSG4LRHBTFTV6PFH44G46H6VTT566U5SYXUEDIA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36006->145.79.227.8:4330: use of closed network connection Nov 21 04:06:55.836176589 (284h2m54.38764707s) [noise=true , address=145.79.227.4:4185 , piece_id=SNLK4Y2VFS7ONUEG3KL6M7UGC4HLKUSVB4DWODQBGAIH6BTS6JVA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36010->145.79.227.4:4185: use of closed network connection Nov 21 04:06:55.836639490 (284h2m54.388109961s) [noise=true , address=145.79.227.20:4438 , piece_id=7X5WCGRDQLESZ5QMPRLVE2LBDRFAGZYWUMODQI67VW6447FNILRA, seg_pos={0 0}, upload=250173]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:53692->145.79.227.20:4438: use of closed network connection Nov 21 04:06:55.836673114 (284h2m54.388143575s) [seg_pos={0 0}, upload=250173]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 04:06:56.079921669 (284h2m54.631392131s) [upload=250173]: Done uploading segments... Nov 21 04:06:56.079966486 (284h2m54.631436937s) [upload=250173]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.247040519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.45023263s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:43:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45027536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.502296095s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:42:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502355858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.400888754s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:41:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400916748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.451106863s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:41:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.451163481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.534025495s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:40:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534120476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.796157379s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.796187676s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.453239763s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:39:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453263599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.904239795s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:39:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904487397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.977034107s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:38:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.977062803s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.559403479s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:38:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559430129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.275394195s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:37:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.275414575s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.532321299s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:37:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532349463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.856980249s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:37:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.857010327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.600680753s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:37:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600723073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.748896562s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.74892233s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.121429775s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:36:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.121686093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.887161268s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:35:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.887253444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.401447728s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:35:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401478798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.064535181s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.064568222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.489775429s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:34:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489804394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.916255969s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:34:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.916284153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.909740716s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:34:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909843792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.517301093s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:34:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517328706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.957629512s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:33:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.957662696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.397659236s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:32:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397694724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.............X.............................................] duration: 3.799054228s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:32:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.799086349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.............X............................................] duration: 3.462961351s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:32:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462993271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.883972886s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:31:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.883999217s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)