Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X....XX............X........X.X.XX....X.X.XX.XX.............] duration: 3.654225715s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:53:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654261183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 8.426066604s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:53:41 UTC) orbiter warning: context deadline exceeded Duration: 8.426096121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 9.666132468s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:53:24 UTC) orbiter warning: context deadline exceeded Duration: 9.666158087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X....XX............X........X.X.XX....X.X.XX.XX............] duration: 3.87383495s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:53:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.873883372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..XX.X......X.X....X.X......X...X....................] duration: 9.638350575s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:53:21 UTC) orbiter warning: context deadline exceeded Duration: 9.638392375s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 12.452289708s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:53:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 12.452313153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 12.031343714s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:52:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 12.031368552s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......X..XX.X......X.X....X.X......X...X...................] duration: 7.247802414s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:52:37 UTC) orbiter warning: context deadline exceeded Duration: 7.247833934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.442096414s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:52:35 UTC) orbiter warning: context deadline exceeded Duration: 7.442127022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X.X....XX............X........X.X.XX....X.X.XX.XX...........] duration: 3.8071556s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:52:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80718185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.14066871s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:51:49 UTC) orbiter warning: context deadline exceeded Duration: 7.140696715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 15.375612307s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:51:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.375653655s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X..XX.X......X.X....X.X......X...X..................] duration: 6.288752404s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:51:38 UTC) orbiter warning: context deadline exceeded Duration: 6.288779837s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X.X....XX............X........X.X.XX....X.X.XX.XX..........] duration: 3.659331368s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:51:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.659358269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.058566497s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:51:11 UTC) orbiter warning: context deadline exceeded Duration: 9.058593619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.329056124s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.329133121s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X......X..XX.X......X.X....X.X......X...X.................] duration: 6.09559514s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:50:51 UTC) orbiter warning: context deadline exceeded Duration: 6.09565315s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X.X....XX............X........X.X.XX....X.X.XX.XX.........] duration: 4.446775349s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:50:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.446802058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 15.096776213s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.096803686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 12.73767984s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:50:06 UTC) orbiter warning: context deadline exceeded Duration: 12.737710448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X......X..XX.X......X.X....X.X......X...X................] duration: 7.626374251s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:50:04 UTC) orbiter warning: context deadline exceeded Duration: 7.626392836s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...X.X....XX............X........X.X.XX....X.X.XX.XX........] duration: 3.531071228s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:49:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531099962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 11.421560429s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:49:50 UTC) orbiter warning: context deadline exceeded Duration: 11.421592842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 14.727794458s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:49:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.727829275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.583523445s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:49:20 UTC) orbiter warning: context deadline exceeded Duration: 11.58354669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X......X..XX.X......X.X....X.X......X...X...............] duration: 7.225735364s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:49:10 UTC) orbiter warning: context deadline exceeded Duration: 7.225939474s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [....X.X....XX............X........X.X.XX....X.X.XX.XX.......] duration: 5.000621468s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:49:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000647028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 14.625052169s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:49:01 UTC) orbiter warning: context deadline exceeded Duration: 14.62507436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.99852416s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:48:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.998556973s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 23.589380528s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:48:37 UTC) orbiter warning: context deadline exceeded Duration: 23.589412549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X....X.X....XX............X........X.X.XX....X.X.XX.XX......] duration: 4.900145074s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:48:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.900166596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X......X..XX.X......X.X....X.X......X...X..............] duration: 8.464640005s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:48:23 UTC) orbiter warning: context deadline exceeded Duration: 8.464669933s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 15.142420226s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:48:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.142450443s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X....X.X....XX............X........X.X.XX....X.X.XX.XX.....] duration: 3.552675304s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:47:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552695783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X......X..XX.X......X.X....X.X......X...X.............] duration: 6.384926927s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:47:40 UTC) orbiter warning: context deadline exceeded Duration: 6.384957297s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.26046691s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:47:37 UTC) orbiter warning: context deadline exceeded Duration: 7.260497078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 14.653210306s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:47:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.653241977s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.57128394s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:47:05 UTC) orbiter warning: context deadline exceeded Duration: 7.57131513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 14.85527868s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:47:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.855334868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X....X.X....XX............X........X.X.XX....X.X.XX.XX....] duration: 4.172650317s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:46:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.172677198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 14.729265549s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.729292339s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X......X..XX.X......X.X....X.X......X...X............] duration: 9.21584984s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:46:40 UTC) orbiter warning: context deadline exceeded Duration: 9.215878925s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X......X..XX.X......X.X....X.X......X...X...........] duration: 15.402533649s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:46:26 UTC) orbiter warning: context deadline exceeded Duration: 15.402562234s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X..X....X.X....XX............X........X.X.XX....X.X.XX.XX...] duration: 5.001810633s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:46:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001840711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 18.498790434s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:46:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 18.498826082s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 9.128627333s trace: https://jaeger.dev.storj.io/trace/3b4f544d20103a60 FAIL (2025-11-22 15:45:52 UTC) orbiter warning: context deadline exceeded Duration: 9.128656148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XX..X....X.X....XX............X........X.X.XX....X.X.XX.XX..] duration: 4.704506894s trace: https://jaeger.dev.storj.io/trace/9f42460335cf8a FAIL (2025-11-22 15:45:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.704537152s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........X......X..XX.X......X.X....X.X......X...X..........] duration: 11.107373311s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:45:41 UTC) orbiter warning: context deadline exceeded Duration: 11.107403851s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 14.891082978s trace: https://jaeger.dev.storj.io/trace/57c052b4cc41de17 FAIL (2025-11-22 15:45:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 2]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 15:19:09.166749739 (319h15m7.718220201s) [upload=281162]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 15:19:09.776915354 (319h15m8.328385805s) [seg_pos={0 0}, upload=281162]: Begin upload segment... Nov 22 15:19:09.776989455 (319h15m8.328459916s) [seg_pos={0 0}, upload=281162]: Done begin upload segment. Nov 22 15:19:09.777068767 (319h15m8.328539248s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777079627 (319h15m8.328550088s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777145903 (319h15m8.328616364s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.777958190 (319h15m8.329428641s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778004879 (319h15m8.32947533s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778414749 (319h15m8.32988521s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778557892 (319h15m8.330028353s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.778823878 (319h15m8.330294339s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.779217277 (319h15m8.330687738s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780002683 (319h15m8.331473164s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.780733865 (319h15m8.332204316s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.781283973 (319h15m8.332754434s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.782212761 (319h15m8.333683222s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.784477244 (319h15m8.335947705s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785183759 (319h15m8.33665422s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:09.785447691 (319h15m8.336918162s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Uploading piece... Nov 22 15:19:12.087675155 (319h15m10.639145616s) [upload=281162]: Next returned nil segment Nov 22 15:19:12.087694182 (319h15m10.639164643s) [upload=281162]: Waiting for error group managing segments... Nov 22 15:19:12.674340982 (319h15m11.225811443s) [noise=true , address=45.91.116.9:4262 , piece_id=KHLLMSTI5AYSNXNETY7VU3Q4QVWHACD5Q4KUOVVPQVFSAEDAD6FQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:12.798613018 (319h15m11.350083479s) [noise=true , address=77.90.31.11:4554 , piece_id=NVF2HJF7TTUSS3BQ5OIP4BV5RXVXLGTZNHIH2KG2PAPVAO3Z6WQA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:20.945773203 (319h15m19.497243664s) [noise=true , address=145.79.227.18:4006 , piece_id=54IKPIXLBSYXPDU2U7V2ASGIGSK7VO6OM6OLSQ3244EA3ZRTH64A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:22.747233456 (319h15m21.298703907s) [noise=true , address=145.79.227.5:4221 , piece_id=B24Q3KWEMTIUUUX4BMEHSYZFRU2YAGZM4LG5MLJPSEZPFSSB2M7Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.261982510 (319h15m21.813452971s) [noise=true , address=145.79.227.20:4439 , piece_id=J6VZR5KBODPJ5X4JQMJKYNLEIG6PD3ZDPPGDSMD66IMZS6UGJWXQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.577334318 (319h15m22.128804879s) [noise=true , address=145.79.227.23:4546 , piece_id=CD5KXVCDZRIFZ562T7OJ352FUG2H3XOU3YWN5DS3BGY2724LOM6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:23.685681951 (319h15m22.237152412s) [noise=true , address=145.79.227.28:4727 , piece_id=IEMFOGUE3INE2XISL4ZSKIAL47OLHWJDUQH5XAYC3KTTEM2Z7Q3Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:30.145657703 (319h15m28.697128165s) [noise=true , address=145.79.227.24:4583 , piece_id=LVRLTM2EI45YKWTJ37YMH5W6T5VVW4EVUFB6ABT452HLUCPR4QEA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: <nil> Nov 22 15:19:37.408487345 (319h15m35.959957816s) [noise=true , address=145.79.227.9:4365 , piece_id=GMS2VOHHJNJYYJK6WNOGMC4WZPQCWEFQGNTNLHFA7TIJ22ITLMAQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->145.79.227.9:4365: use of closed network connection Nov 22 15:19:37.408552439 (319h15m35.96002291s) [noise=true , address=145.79.227.5:4223 , piece_id=GPCMRZKPRDTGTEQPHXN4JRBFJMFSTICA2J67QFANZNZSMTMKAF6A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.408919488 (319h15m35.960389949s) [noise=true , address=145.79.227.21:4475 , piece_id=IIB3HW45LECRXWEPBZNA4DZUD2CJDU2BJIBEXY2J5WDRDSVUEU6Q, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:54040->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409052922 (319h15m35.960523393s) [noise=true , address=145.79.227.32:4834 , piece_id=F355YDI324R46UBK5R3NUGDX2MVNE7EP4E7CAGZVXWE4DVNGTYQQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:49164->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409079863 (319h15m35.960550345s) [noise=true , address=145.79.227.30:4762 , piece_id=2SLJEV2VEWM2JSODZVZS2GDKQBSLRJCJ3IB277O3JYGVJVOFKS4A, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 22 15:19:37.409200603 (319h15m35.960671064s) [noise=true , address=145.79.227.20:4437 , piece_id=KK3XHGQJ4YXZHX7FPPVTKM36K3SFBTCOSQAGVXBKS4RETX6SWPHQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->145.79.227.20:4437: use of closed network connection Nov 22 15:19:37.409299511 (319h15m35.960769972s) [noise=true , address=145.79.227.26:4653 , piece_id=Y3VHA667RADVZA2IYHLJZFMAURMRMABFFN24ROZMBUX5YDHJCLVA, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:43938->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409530461 (319h15m35.961000953s) [noise=true , address=145.79.227.9:4367 , piece_id=ZG4ANHMI7VN3I7JPRFJPR3KK4QEPN27D2KZRE725TXPIUJ5YBEMQ, seg_pos={0 0}, upload=281162]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:55550->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 15:19:37.409609351 (319h15m35.961079813s) [seg_pos={0 0}, upload=281162]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 15:19:37.945679996 (319h15m36.497150457s) [upload=281162]: Done uploading segments... Nov 22 15:19:37.945726795 (319h15m36.497197246s) [upload=281162]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.891112084s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X......X..XX.X......X.X....X.X......X...X.........] duration: 9.289651261s trace: https://jaeger.dev.storj.io/trace/539f65bbf9281c09 FAIL (2025-11-22 15:44:53 UTC) orbiter warning: context deadline exceeded Duration: 9.289679755s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)