Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.13240509s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:54:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 7.132435998s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.566177999s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:53:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.566232944s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..XX.X..X..............X...X.............X...XX.............] duration: 3.67946791s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:53:34 UTC) orbiter warning: context deadline exceeded Duration: 3.679492267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.792618815s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:53:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.792648993s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.038058234s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:53:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.038085887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.713961641s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:52:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.713989482s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 14.137098393s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:52:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.137155341s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...XX.X..X..............X...X.............X...XX............] duration: 3.38695092s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:52:23 UTC) orbiter warning: context deadline exceeded Duration: 3.386972091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....XX.X..X..............X...X.............X...XX...........] duration: 3.672214428s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:52:06 UTC) orbiter warning: context deadline exceeded Duration: 3.672239786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.983816124s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:52:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.983844338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 12.739211981s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:52:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.739268678s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.833224014s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:51:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.833253251s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.893725445s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:51:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.893760482s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....XX.X..X..............X...X.............X...XX..........] duration: 3.66258484s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:50:52 UTC) orbiter warning: context deadline exceeded Duration: 3.662655165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.804560868s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.80458852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......XX.X..X..............X...X.............X...XX.........] duration: 3.763981281s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:50:28 UTC) orbiter warning: context deadline exceeded Duration: 3.764016578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 14.061833224s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:50:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.061854434s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......XX.X..X..............X...X.............X...XX........] duration: 3.394066427s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:49:59 UTC) orbiter warning: context deadline exceeded Duration: 3.394093269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.879245969s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:49:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.879272491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........XX.X..X..............X...X.............X...XX.......] duration: 3.686826461s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:49:26 UTC) orbiter warning: context deadline exceeded Duration: 3.686892918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.896374791s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:49:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.896409036s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.837491134s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:48:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.8375196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X........XX.X..X..............X...X.............X...XX......] duration: 3.501292639s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:48:46 UTC) orbiter warning: context deadline exceeded Duration: 3.50131951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.780256885s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:48:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.780304256s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.880518257s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:48:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.880550108s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X........XX.X..X..............X...X.............X...XX.....] duration: 3.743768665s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:48:15 UTC) orbiter warning: context deadline exceeded Duration: 3.743798572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 16.15306969s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:47:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.153124435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.94844326s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:47:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.948472304s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X........XX.X..X..............X...X.............X...XX....] duration: 3.689976939s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:47:34 UTC) orbiter warning: context deadline exceeded Duration: 3.690005732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X........XX.X..X..............X...X.............X...XX...] duration: 3.691207044s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:47:16 UTC) orbiter warning: context deadline exceeded Duration: 3.691224127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 15.379315242s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:46:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.379411415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................................X.............] duration: 5.822927771s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:46:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.822953149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................................X............] duration: 5.781296711s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:46:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.781326978s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X........XX.X..X..............X...X.............X...XX..] duration: 3.663767153s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:46:24 UTC) orbiter warning: context deadline exceeded Duration: 3.663800007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................................X...........] duration: 5.812883491s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:46:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.812924829s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 18.123263111s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:46:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.123290693s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X........XX.X..X..............X...X.............X...XX.] duration: 3.669912495s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:46:02 UTC) orbiter warning: context deadline exceeded Duration: 3.669943533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................................X..........] duration: 5.849012217s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.849084533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................................X.........] duration: 5.792524948s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:45:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.792558293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X........XX.X..X..............X...X.............X...XX] duration: 5.000871343s trace: https://jaeger.dev.storj.io/trace/53f044530c43732e FAIL (2025-11-30 14:45:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001036719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................................X........] duration: 5.877280295s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.877309732s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.135466321s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:44:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.135486771s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X........XX.X..X..............X...X.............X...X] duration: 5.001675605s trace: https://jaeger.dev.storj.io/trace/167a4d97771c6d5c FAIL (2025-11-30 14:44:43 UTC) orbiter warning: context deadline exceeded Duration: 5.00169924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 12.768886257s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:44:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.768921404s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................................X.......] duration: 5.792446121s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.792475227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.......X........XX.X..X..............X...X.............X...] duration: 3.486921754s trace: https://jaeger.dev.storj.io/trace/50c318c6e7b80f04 FAIL (2025-11-30 14:43:27 UTC) orbiter warning: context deadline exceeded Duration: 3.486978923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................................X......] duration: 5.831555947s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:43:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.831596695s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.243869003s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:42:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.243895935s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.......X........XX.X..X..............X...X.............X..] duration: 3.699731082s trace: https://jaeger.dev.storj.io/trace/50c318c6e7b80f04 FAIL (2025-11-30 14:42:56 UTC) orbiter warning: context deadline exceeded Duration: 3.69975653s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.......X........XX.X..X..............X...X.............X.] duration: 3.386846592s trace: https://jaeger.dev.storj.io/trace/50c318c6e7b80f04 FAIL (2025-11-30 14:42:40 UTC) orbiter warning: context deadline exceeded Duration: 3.386876889s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)