Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X...XX..............X.........X.......X............] duration: 3.452525549s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:18:26 UTC) orbiter warning: context deadline exceeded Duration: 3.452550346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.769107134s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.769128083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X...XX..............X.........X.......X...........] duration: 3.749819922s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:17:26 UTC) orbiter warning: context deadline exceeded Duration: 3.749847635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.793996754s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:17:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.794020951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X...XX..............X.........X.......X..........] duration: 3.749100213s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:16:47 UTC) orbiter warning: context deadline exceeded Duration: 3.749190625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X...XX..............X.........X.......X.........] duration: 4.406251456s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:16:32 UTC) orbiter warning: context deadline exceeded Duration: 4.406278468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.844701479s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.844728741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.829911255s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.829943376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X...XX..............X.........X.......X........] duration: 3.648684765s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:15:28 UTC) orbiter warning: context deadline exceeded Duration: 3.648712788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.............X...XX..............X.........X.......X.......] duration: 3.683318603s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:14:26 UTC) orbiter warning: context deadline exceeded Duration: 3.683343319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.748839908s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.748873782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.............X...XX..............X.........X.......X......] duration: 3.523524212s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:13:34 UTC) orbiter warning: context deadline exceeded Duration: 3.523553957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.849901089s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:13:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.849960884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.811452457s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:13:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.811494076s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.............X...XX..............X.........X.......X.....] duration: 3.521600778s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:12:44 UTC) orbiter warning: context deadline exceeded Duration: 3.521633332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.766574868s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.766604303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.849530705s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:11:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.849560621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.............X...XX..............X.........X.......X....] duration: 3.358069971s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:11:47 UTC) orbiter warning: context deadline exceeded Duration: 3.358100758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.815398007s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:11:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.815427713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X...X.............X...XX..............X.........X.......X...] duration: 3.46971481s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:11:16 UTC) orbiter warning: context deadline exceeded Duration: 3.469743122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.761589002s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:10:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.761616925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X...X.............X...XX..............X.........X.......X..] duration: 3.563931202s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:10:27 UTC) orbiter warning: context deadline exceeded Duration: 3.563966549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.069195168s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.06922762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X...X.............X...XX..............X.........X.......X.] duration: 3.705004497s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:09:13 UTC) orbiter warning: context deadline exceeded Duration: 3.705031008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.135509848s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:09:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.135544434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.962470685s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:08:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.962501333s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X...X.............X...XX..............X.........X.......X] duration: 5.001665122s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:08:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001689577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.889078279s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.889106864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X...X.............X...XX..............X.........X.......] duration: 3.815614809s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:07:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815661097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.794165439s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:07:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.794196597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X...X.............X...XX..............X.........X......] duration: 3.672731287s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:07:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.672764542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X...X.............X...XX..............X.........X.....] duration: 3.989754194s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:06:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.98978318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.927953216s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:06:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.927983553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X...X.............X...XX..............X.........X....] duration: 3.41737956s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:05:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41741078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.075618399s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.075657293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X...X.............X...XX..............X.........X...] duration: 3.392711697s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:04:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392769348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X...X.............X...XX..............X.........X..] duration: 3.521359632s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:04:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521393936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.868365637s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.868399572s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X...X.............X...XX..............X.........X.] duration: 3.463572409s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:03:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463601033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.793480016s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:02:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.79350793s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........X...X.............X...XX..............X.........X] duration: 5.000762653s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:02:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000808991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.783462584s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.78349304s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X...X.............X...XX..............X.........] duration: 3.695832531s trace: https://jaeger.dev.storj.io/trace/1f76d5546712c330 FAIL (2025-11-30 15:01:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.695861307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X...X.............X...XX..............X........] duration: 3.460527723s trace: https://jaeger.dev.storj.io/trace/1f76d5546712c330 FAIL (2025-11-30 15:01:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460561076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.826821276s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:00:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.826849499s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............X...X.............X...XX..............X.......] duration: 3.550446969s trace: https://jaeger.dev.storj.io/trace/1f76d5546712c330 FAIL (2025-11-30 15:00:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550479932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.792950061s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:00:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.792980338s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..............X...X.............X...XX..............X......] duration: 3.681181244s trace: https://jaeger.dev.storj.io/trace/1f76d5546712c330 FAIL (2025-11-30 14:59:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681208465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X..............X...X.............X...XX..............X.....] duration: 3.679108605s trace: https://jaeger.dev.storj.io/trace/1f76d5546712c330 FAIL (2025-11-30 14:59:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.67913734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.778048489s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:58:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.778078756s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)