Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.X.........X........X............................X.....] duration: 14.976522664s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:23:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.976551481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........XX.XX.................XX..X....X.X....XX............] duration: 3.58061322s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:22:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5806447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......X............X.................X......X..XX.X......X] duration: 30.001475958s trace: https://jaeger.dev.storj.io/trace/73f903c3f116d7f2 FAIL (2025-11-22 15:22:57 UTC) orbiter warning: context deadline exceeded Duration: 30.001500816s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.X.........X........X............................X....] duration: 14.922578685s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:22:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.922611768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.X.........X........X............................X...] duration: 14.03105833s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:22:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.031080182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........XX.XX.................XX..X....X.X....XX...........] duration: 3.569652709s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:22:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569681815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......X............X.................X......X..XX.X......] duration: 28.23291913s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:21:58 UTC) orbiter warning: context deadline exceeded Duration: 28.232936023s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X......X............X.................X......X..XX.X.....] duration: 27.405285248s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:21:29 UTC) orbiter warning: context deadline exceeded Duration: 27.405326087s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.X.........X........X............................X..] duration: 15.099955898s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:21:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.09998872s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.........XX.XX.................XX..X....X.X....XX..........] duration: 3.589071075s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:21:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589102416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X......X............X.................X......X..XX.X....] duration: 22.58798455s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:20:51 UTC) orbiter warning: context deadline exceeded Duration: 22.588111333s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X.........XX.XX.................XX..X....X.X....XX.........] duration: 3.698768578s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:20:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.698856967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.X.........X........X............................X.] duration: 14.886818844s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:20:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.886849531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X.X.........XX.XX.................XX..X....X.X....XX........] duration: 3.509297774s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:20:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509329164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X............X.................X......X..XX.X...] duration: 27.815298896s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:20:15 UTC) orbiter warning: context deadline exceeded Duration: 27.815315287s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X......X............X.................X......X..XX.X..] duration: 29.275243617s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:19:47 UTC) orbiter warning: context deadline exceeded Duration: 29.275276028s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X.X.........XX.XX.................XX..X....X.X....XX.......] duration: 3.786828269s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:19:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.786872564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.X.........X........X............................X] duration: 30.539981573s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:19:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.540005439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X.X.........XX.XX.................XX..X....X.X....XX......] duration: 3.794429874s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:19:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.794469831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X..X.X.........XX.XX.................XX..X....X.X....XX.....] duration: 3.472156141s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:18:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47218806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX.X.........X........X............................] duration: 18.963256851s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:18:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 18.963311466s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X......X............X.................X......X..XX.X.] duration: 29.146617746s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:18:34 UTC) orbiter warning: context deadline exceeded Duration: 29.146650227s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX..X.X.........XX.XX.................XX..X....X.X....XX....] duration: 3.752771359s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:18:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.752800976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XX..X.X.........XX.XX.................XX..X....X.X....XX...] duration: 3.562772628s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:17:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562800432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX.X.........X........X...........................] duration: 14.662405991s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:17:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.662478389s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.......X......X............X.................X......X..XX.X] duration: 30.001091048s trace: https://jaeger.dev.storj.io/trace/51715da3ee2c2f02 FAIL (2025-11-22 15:17:21 UTC) orbiter warning: context deadline exceeded Duration: 30.001109443s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..XX..X.X.........XX.XX.................XX..X....X.X....XX..] duration: 4.131327618s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:16:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.131357215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.......X......X............X.................X......X..XX.] duration: 23.017566212s trace: https://jaeger.dev.storj.io/trace/22495c0fb7d8280d FAIL (2025-11-22 15:16:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.017598483s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX.X.........X........X..........................] duration: 15.584918449s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.584951762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X..XX..X.X.........XX.XX.................XX..X....X.X....XX.] duration: 3.609735594s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:16:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.609767244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX.X.........X........X.........................] duration: 15.042366994s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:15:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.042394496s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.......X......X............X.................X......X..XX] duration: 30.002005418s trace: https://jaeger.dev.storj.io/trace/22495c0fb7d8280d FAIL (2025-11-22 15:15:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002028852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX.X.........X........X........................] duration: 15.381283442s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:15:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.381315894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X..XX..X.X.........XX.XX.................XX..X....X.X....XX] duration: 5.001416394s trace: https://jaeger.dev.storj.io/trace/6c14636bed72b6b9 FAIL (2025-11-22 15:15:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001435621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XX.X.........X........X.......................] duration: 15.184907086s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:14:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.184925691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X..XX..X.X.........XX.XX.................XX..X....X.X....X] duration: 5.001470549s trace: https://jaeger.dev.storj.io/trace/3e6a1066cb0bef7f FAIL (2025-11-22 15:14:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00149284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.......X......X............X.................X......X..X] duration: 30.001276903s trace: https://jaeger.dev.storj.io/trace/210ceb8e1f0aab92 FAIL (2025-11-22 15:14:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001303194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............XX.X.........X........X......................] duration: 14.405345652s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:13:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.405373686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X..X..XX..X.X.........XX.XX.................XX..X....X.X....] duration: 3.685297397s trace: https://jaeger.dev.storj.io/trace/46388904967d5e33 FAIL (2025-11-22 15:13:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.685374102s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.......X......X............X.................X......X..] duration: 21.265016594s trace: https://jaeger.dev.storj.io/trace/4c651f46f6510722 FAIL (2025-11-22 15:13:44 UTC) orbiter warning: context deadline exceeded Duration: 21.265062712s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XX..X..XX..X.X.........XX.XX.................XX..X....X.X...] duration: 3.697371268s trace: https://jaeger.dev.storj.io/trace/46388904967d5e33 FAIL (2025-11-22 15:13:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.69740381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.......X......X............X.................X......X.] duration: 21.045647306s trace: https://jaeger.dev.storj.io/trace/4c651f46f6510722 FAIL (2025-11-22 15:12:49 UTC) orbiter warning: context deadline exceeded Duration: 21.045680779s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................XX.X.........X........X.....................] duration: 15.704480748s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:12:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.704517546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.XX..X..XX..X.X.........XX.XX.................XX..X....X.X..] duration: 3.688161551s trace: https://jaeger.dev.storj.io/trace/46388904967d5e33 FAIL (2025-11-22 15:12:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.688183534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..XX..X..XX..X.X.........XX.XX.................XX..X....X.X.] duration: 3.627270953s trace: https://jaeger.dev.storj.io/trace/46388904967d5e33 FAIL (2025-11-22 15:12:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.627290332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.......X......X............X.................X......X] duration: 30.001528133s trace: https://jaeger.dev.storj.io/trace/4c651f46f6510722 FAIL (2025-11-22 15:12:15 UTC) orbiter warning: context deadline exceeded Duration: 30.00155269s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................XX.X.........X........X....................] duration: 14.607730728s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:11:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.607759442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...XX..X..XX..X.X.........XX.XX.................XX..X....X.X] duration: 5.001720602s trace: https://jaeger.dev.storj.io/trace/46388904967d5e33 FAIL (2025-11-22 15:11:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001742424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X.......X......X............X.................X......] duration: 20.717873585s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:11:09 UTC) orbiter warning: context deadline exceeded Duration: 20.71790194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XX.X.........X........X...................] duration: 16.358482685s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 16.358514837s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)