Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 33.381917206s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:40:13 UTC) orbiter warning: context deadline exceeded Duration: 33.381944638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................X.........................] duration: 15.845098392s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.845153416s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................X......X..XX.X......X.X....X.X......X...X..] duration: 29.044113226s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:39:49 UTC) orbiter warning: context deadline exceeded Duration: 29.044148051s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......XX..X....X.X....XX............X........X.X.XX....X.X.] duration: 3.47995137s trace: https://jaeger.dev.storj.io/trace/66c4a093268bd6da FAIL (2025-11-22 15:39:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479982017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.................X......X..XX.X......X.X....X.X......X...X.] duration: 23.889874699s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:39:15 UTC) orbiter warning: context deadline exceeded Duration: 23.889903535s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 21.988674771s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:39:04 UTC) orbiter warning: context deadline exceeded Duration: 21.989129648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............................X........................] duration: 15.160273415s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:38:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.160300448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........XX..X....X.X....XX............X........X.X.XX....X.X] duration: 5.000958042s trace: https://jaeger.dev.storj.io/trace/66c4a093268bd6da FAIL (2025-11-22 15:38:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000981557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.347946801s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:38:33 UTC) orbiter warning: context deadline exceeded Duration: 7.348003018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.................X......X..XX.X......X.X....X.X......X...X] duration: 30.006861687s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:38:24 UTC) orbiter warning: context deadline exceeded Duration: 30.006906933s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........XX..X....X.X....XX............X........X.X.XX....X.] duration: 4.197100937s trace: https://jaeger.dev.storj.io/trace/7e1dd42a3e6882ba FAIL (2025-11-22 15:38:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.197133478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................X.......................] duration: 18.708707516s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 18.70874002s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 17.789985816s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:37:51 UTC) orbiter warning: context deadline exceeded Duration: 17.790013179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............................X......................] duration: 15.906300071s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:37:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.906329587s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........XX..X....X.X....XX............X........X.X.XX....X] duration: 5.023343921s trace: https://jaeger.dev.storj.io/trace/7e1dd42a3e6882ba FAIL (2025-11-22 15:37:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.023369631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........XX..X....X.X....XX............X........X.X.XX....] duration: 3.749115255s trace: https://jaeger.dev.storj.io/trace/d23b112fbfb077 FAIL (2025-11-22 15:36:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.749143378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.................X......X..XX.X......X.X....X.X......X...] duration: 7.705858168s trace: https://jaeger.dev.storj.io/trace/6710dd0bac1bedcf FAIL (2025-11-22 15:36:52 UTC) orbiter warning: context deadline exceeded Duration: 7.705885951s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X........X............................X.....................] duration: 15.769314074s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:36:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.769343652s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.581636461s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:36:34 UTC) orbiter warning: context deadline exceeded Duration: 6.581664245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.................X......X..XX.X......X.X....X.X......X..] duration: 7.471588096s trace: https://jaeger.dev.storj.io/trace/6710dd0bac1bedcf FAIL (2025-11-22 15:36:25 UTC) orbiter warning: context deadline exceeded Duration: 7.471605299s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.836227308s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:36:11 UTC) orbiter warning: context deadline exceeded Duration: 6.83624896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X........X............................X....................] duration: 15.062601047s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:35:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.062632847s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.118566169s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:35:52 UTC) orbiter warning: context deadline exceeded Duration: 7.118593853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............XX..X....X.X....XX............X........X.X.XX...] duration: 3.605541436s trace: https://jaeger.dev.storj.io/trace/d23b112fbfb077 FAIL (2025-11-22 15:35:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605571392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.................X......X..XX.X......X.X....X.X......X.] duration: 6.10073911s trace: https://jaeger.dev.storj.io/trace/6710dd0bac1bedcf FAIL (2025-11-22 15:35:12 UTC) orbiter warning: context deadline exceeded Duration: 6.100772543s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.860903225s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:35:00 UTC) orbiter warning: context deadline exceeded Duration: 6.860930817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X........X............................X...................] duration: 15.613819135s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:34:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.613917944s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.................X......X..XX.X......X.X....X.X......X] duration: 30.001468148s trace: https://jaeger.dev.storj.io/trace/6710dd0bac1bedcf FAIL (2025-11-22 15:34:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001493506s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.............XX..X....X.X....XX............X........X.X.XX..] duration: 3.706300277s trace: https://jaeger.dev.storj.io/trace/d23b112fbfb077 FAIL (2025-11-22 15:34:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.706327158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X........X............................X..................] duration: 15.375960982s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:34:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.375993353s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X.................X......X..XX.X......X.X....X.X......] duration: 25.215745469s trace: https://jaeger.dev.storj.io/trace/1ecd3bb44d79eef2 FAIL (2025-11-22 15:34:06 UTC) orbiter warning: context deadline exceeded Duration: 25.215795654s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..............XX..X....X.X....XX............X........X.X.XX.] duration: 3.504574406s trace: https://jaeger.dev.storj.io/trace/d23b112fbfb077 FAIL (2025-11-22 15:34:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504607598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.615484696s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:33:55 UTC) orbiter warning: context deadline exceeded Duration: 6.615517107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...............XX..X....X.X....XX............X........X.X.XX] duration: 5.002074718s trace: https://jaeger.dev.storj.io/trace/d23b112fbfb077 FAIL (2025-11-22 15:33:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00210144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 24.715922249s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:33:43 UTC) orbiter warning: context deadline exceeded Duration: 24.715953779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X........X............................X.................] duration: 14.547075206s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:33:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.547109051s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [................XX..X....X.X....XX............X........X.X.X] duration: 5.001263754s trace: https://jaeger.dev.storj.io/trace/132ae7a30f8d0f8a FAIL (2025-11-22 15:33:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001291438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X.................X......X..XX.X......X.X....X.X.....] duration: 6.855313902s trace: https://jaeger.dev.storj.io/trace/1ecd3bb44d79eef2 FAIL (2025-11-22 15:33:21 UTC) orbiter warning: context deadline exceeded Duration: 6.855345432s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X........X............................X................] duration: 15.358980588s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.359009262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X........X............................X...............] duration: 15.65102858s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.651059639s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X.................X......X..XX.X......X.X....X.X....] duration: 10.699709628s trace: https://jaeger.dev.storj.io/trace/1ecd3bb44d79eef2 FAIL (2025-11-22 15:32:45 UTC) orbiter warning: context deadline exceeded Duration: 10.699767498s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.264286563s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:32:38 UTC) orbiter warning: context deadline exceeded Duration: 6.264317631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................XX..X....X.X....XX............X........X.X.] duration: 3.716924818s trace: https://jaeger.dev.storj.io/trace/4beba22cff9a9db8 FAIL (2025-11-22 15:32:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.716959133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X........X............................X..............] duration: 14.509145452s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.509180539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.112007478s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:31:53 UTC) orbiter warning: context deadline exceeded Duration: 6.112037555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X.................X......X..XX.X......X.X....X.X...] duration: 7.816531951s trace: https://jaeger.dev.storj.io/trace/1ecd3bb44d79eef2 FAIL (2025-11-22 15:31:49 UTC) orbiter warning: context deadline exceeded Duration: 7.81656857s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X.................X......X..XX.X......X.X....X.X..] duration: 28.017457322s trace: https://jaeger.dev.storj.io/trace/1ecd3bb44d79eef2 FAIL (2025-11-22 15:31:34 UTC) orbiter warning: context deadline exceeded Duration: 28.017488931s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 14.783945871s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:31:22 UTC) orbiter warning: context deadline exceeded Duration: 14.78397171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.................XX..X....X.X....XX............X........X.X] duration: 5.000605291s trace: https://jaeger.dev.storj.io/trace/4beba22cff9a9db8 FAIL (2025-11-22 15:31:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000848394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X........X............................X.............] duration: 14.734131257s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.734176272s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)