Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X......X.X....X.X......X...X................................] duration: 7.16365904s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:02:57 UTC) orbiter warning: context deadline exceeded Duration: 7.163687866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 14.716735619s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 16:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.716755106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 18.715230888s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:02:47 UTC) orbiter warning: context deadline exceeded Duration: 18.715302424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 14.459318832s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 16:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.459347918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [......X........X.X.XX....X.X.XX.XX...................XXX.XX.] duration: 3.372117268s trace: https://jaeger.dev.storj.io/trace/4c5b769530e50f21 FAIL (2025-11-22 16:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.372148186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X......X.X....X.X......X...X...............................] duration: 19.639510076s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:02:16 UTC) orbiter warning: context deadline exceeded Duration: 19.639602562s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.415208451s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:02:10 UTC) orbiter warning: context deadline exceeded Duration: 6.415236884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.......X........X.X.XX....X.X.XX.XX...................XXX.XX] duration: 5.002195619s trace: https://jaeger.dev.storj.io/trace/4c5b769530e50f21 FAIL (2025-11-22 16:02:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002224264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 15.573862157s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 16:01:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.573895981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.663097214s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:01:18 UTC) orbiter warning: context deadline exceeded Duration: 6.66312679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [........X........X.X.XX....X.X.XX.XX...................XXX.X] duration: 5.001378964s trace: https://jaeger.dev.storj.io/trace/e41c7b1745e33e FAIL (2025-11-22 16:01:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001403601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X......X.X....X.X......X...X..............................] duration: 14.063151689s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:01:03 UTC) orbiter warning: context deadline exceeded Duration: 14.063188981s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.402399683s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 16:00:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.402434049s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.........X........X.X.XX....X.X.XX.XX...................XXX.] duration: 3.485298411s trace: https://jaeger.dev.storj.io/trace/24d1bb76658157a2 FAIL (2025-11-22 16:00:44 UTC) orbiter warning: context deadline exceeded Duration: 3.485494756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.X......X.X....X.X......X...X.............................] duration: 16.556947781s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 16:00:23 UTC) orbiter warning: context deadline exceeded Duration: 16.556976907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 16.492929149s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 16:00:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 16.492951461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.089765425s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 16:00:04 UTC) orbiter warning: context deadline exceeded Duration: 7.089797214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX.X......X.X....X.X......X...X............................] duration: 17.771395487s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:59:53 UTC) orbiter warning: context deadline exceeded Duration: 17.771426176s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..........X........X.X.XX....X.X.XX.XX...................XXX] duration: 5.001872672s trace: https://jaeger.dev.storj.io/trace/24d1bb76658157a2 FAIL (2025-11-22 15:59:35 UTC) orbiter warning: context deadline exceeded Duration: 5.001892569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.517998871s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:59:28 UTC) orbiter warning: context deadline exceeded Duration: 6.518027705s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX.X......X.X....X.X......X...X...........................] duration: 17.909404039s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:59:03 UTC) orbiter warning: context deadline exceeded Duration: 17.909431511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 12.338221853s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 12.338265386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.468929509s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:58:33 UTC) orbiter warning: context deadline exceeded Duration: 7.468973983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...........X........X.X.XX....X.X.XX.XX...................XX] duration: 5.00065371s trace: https://jaeger.dev.storj.io/trace/20a5ec32ada9f4e6 FAIL (2025-11-22 15:58:22 UTC) orbiter warning: context deadline exceeded Duration: 5.0006799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 11.549054921s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:58:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 11.549165772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.688043254s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:58:08 UTC) orbiter warning: context deadline exceeded Duration: 6.688075003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............X........X.X.XX....X.X.XX.XX...................X] duration: 5.001485114s trace: https://jaeger.dev.storj.io/trace/200c1e004f5ebfa7 FAIL (2025-11-22 15:57:56 UTC) orbiter warning: context deadline exceeded Duration: 5.00152006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..XX.X......X.X....X.X......X...X..........................] duration: 7.160842557s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:57:54 UTC) orbiter warning: context deadline exceeded Duration: 7.160871593s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..XX.X......X.X....X.X......X...X.........................] duration: 14.405316466s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:57:36 UTC) orbiter warning: context deadline exceeded Duration: 14.405333718s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X............X........X.X.XX....X.X.XX.XX...................] duration: 4.849316867s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:57:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.849346414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 14.743951585s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:57:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.744001089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 14.826107122s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:57:24 UTC) orbiter warning: context deadline exceeded Duration: 14.826173359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..XX.X......X.X....X.X......X...X........................] duration: 18.384291439s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:57:21 UTC) orbiter warning: context deadline exceeded Duration: 18.384336514s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 14.758757126s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:56:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.75878554s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XX............X........X.X.XX....X.X.XX.XX..................] duration: 3.833491848s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:56:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.833718291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..XX.X......X.X....X.X......X...X.......................] duration: 7.904356738s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:56:29 UTC) orbiter warning: context deadline exceeded Duration: 7.904372499s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX............X........X.X.XX....X.X.XX.XX.................] duration: 4.107314305s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:56:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.107407002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 20.089066828s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:56:19 UTC) orbiter warning: context deadline exceeded Duration: 20.08909931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 16.527969065s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:56:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 16.527988834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX............X........X.X.XX....X.X.XX.XX................] duration: 4.075680839s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:55:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.075712199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.067552285s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.067578447s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX............X........X.X.XX....X.X.XX.XX...............] duration: 3.762185749s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:55:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.762220827s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..XX.X......X.X....X.X......X...X......................] duration: 7.605757553s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:55:34 UTC) orbiter warning: context deadline exceeded Duration: 7.605788511s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.730889841s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:55:29 UTC) orbiter warning: context deadline exceeded Duration: 6.730942903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.967762844s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:55:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 13.967791407s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....XX............X........X.X.XX....X.X.XX.XX..............] duration: 3.508224943s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:55:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508254089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.313303251s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:55:02 UTC) orbiter warning: context deadline exceeded Duration: 6.313416807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.678763745s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:54:37 UTC) orbiter warning: context deadline exceeded Duration: 6.678809531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 11.538932647s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:54:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 11.538982722s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..XX.X......X.X....X.X......X...X.....................] duration: 8.186770514s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:54:22 UTC) orbiter warning: context deadline exceeded Duration: 8.186783969s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)