Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............................................X....] duration: 5.86627687s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:42:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.866305155s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.......X........XX.X..X..............X...X.............X] duration: 5.001428077s trace: https://jaeger.dev.storj.io/trace/50c318c6e7b80f04 FAIL (2025-11-30 14:42:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001454719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............................................X...] duration: 6.036727253s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.03677297s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 13.569277641s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:41:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.569311696s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X.......X........XX.X..X..............X...X.............] duration: 3.466520065s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:40:50 UTC) orbiter warning: context deadline exceeded Duration: 3.466550833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............................................X..] duration: 5.872176173s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.872212471s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 9.939754418s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:40:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.939790989s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X.......X........XX.X..X..............X...X............] duration: 3.639905977s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:39:58 UTC) orbiter warning: context deadline exceeded Duration: 3.640197993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.004353781s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:39:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.004382847s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............................................X.] duration: 5.776671414s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:39:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.776699306s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 12.609150545s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:39:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.609178498s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................................X] duration: 30.257760935s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 14:39:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 30.257779071s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X.......X........XX.X..X..............X...X...........] duration: 3.477269675s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:38:49 UTC) orbiter warning: context deadline exceeded Duration: 3.477298691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.048141078s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:38:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.048174761s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 13.76667058s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:38:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.766698454s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X.......X........XX.X..X..............X...X..........] duration: 3.3961521s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:38:19 UTC) orbiter warning: context deadline exceeded Duration: 3.39617312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 12.277800136s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:38:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.278228532s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........X.......X........XX.X..X..............X...X.........] duration: 3.689613929s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:37:42 UTC) orbiter warning: context deadline exceeded Duration: 3.68965604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.822130749s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:37:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.822155856s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 11.671310673s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:37:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.671338556s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.848598614s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:36:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.848626287s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........X.......X........XX.X..X..............X...X........] duration: 3.724117745s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:36:42 UTC) orbiter warning: context deadline exceeded Duration: 3.724145778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 14.702339249s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:36:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.702368663s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 13.792865461s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:36:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.793120727s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.741169539s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:36:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.741202229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........X.......X........XX.X..X..............X...X.......] duration: 3.781151192s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:36:05 UTC) orbiter warning: context deadline exceeded Duration: 3.78118632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.876180928s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:35:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.876213609s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 9.982805329s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:35:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.982845826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.919234211s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:35:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.919255541s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 19.749079816s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:34:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.749103231s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........X.......X........XX.X..X..............X...X......] duration: 3.509743081s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:34:57 UTC) orbiter warning: context deadline exceeded Duration: 3.509772497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.839050034s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:34:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.839108115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............X.......X........XX.X..X..............X...X.....] duration: 3.462337832s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:34:02 UTC) orbiter warning: context deadline exceeded Duration: 3.462369222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.865991216s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:33:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.866075076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.337627042s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:33:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.337655226s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 15.482473081s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:33:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.48249304s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............X.......X........XX.X..X..............X...X....] duration: 3.636683857s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:33:20 UTC) orbiter warning: context deadline exceeded Duration: 3.636715778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.747655301s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:32:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.747681241s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............X.......X........XX.X..X..............X...X...] duration: 3.554237787s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:32:32 UTC) orbiter warning: context deadline exceeded Duration: 3.554267404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.213212405s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:32:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.213241963s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 22.579417153s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:31:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.579438803s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.85789747s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.857926174s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............X.......X........XX.X..X..............X...X..] duration: 3.570535774s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:31:27 UTC) orbiter warning: context deadline exceeded Duration: 3.57056518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 13.96398489s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:31:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.964004567s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.74093712s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:30:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.740966746s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................X.......X........XX.X..X..............X...X.] duration: 3.53729342s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:30:29 UTC) orbiter warning: context deadline exceeded Duration: 3.537321765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.873429713s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:30:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.873453167s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.097937971s trace: https://jaeger.dev.storj.io/trace/39ff5001d414bcdf FAIL (2025-11-30 14:30:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.098003565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.812975255s trace: https://jaeger.dev.storj.io/trace/657a298d72a1dcb7 FAIL (2025-11-30 14:29:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 582s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822498353 (510h2m14.373968814s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.822919635 (510h2m14.374390106s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823149052 (510h2m14.374619513s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.823439715 (510h2m14.374910166s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:15.824058944 (510h2m14.375529405s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Uploading piece... Nov 30 14:06:17.641631834 (510h2m16.193102295s) [upload=449157]: Next returned nil segment Nov 30 14:06:17.641649628 (510h2m16.193120089s) [upload=449157]: Waiting for error group managing segments... Nov 30 14:06:17.961591160 (510h2m16.513061621s) [noise=true , address=45.91.116.6:4973 , piece_id=YZIPW5BAFLW6B3K5FYEA2TUQKGRNYMNWXP36DKYBRX3OOHQERTYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.985403504 (510h2m16.536873965s) [noise=true , address=45.91.116.13:4153 , piece_id=2F2NA4ZKZLKXZZM3IROT4OTLJI5KEJCBLQZJIGMSS7YI5XYGVREA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:17.988791976 (510h2m16.540262427s) [noise=true , address=45.91.116.5:4298 , piece_id=JOWT7R3PLXA2CSXIPTKSESLEXQNZHZRJYF7LNNTJWVNVYS4XUMYQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: <nil> Nov 30 14:06:44.864740849 (510h2m43.41621131s) [noise=true , address=145.79.227.30:4762 , piece_id=HCB4MXJFWV5N77RMKDGNA2KYLY7IEB2F57JBITUOQ5I3OX6O6WIA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->145.79.227.30:4762: use of closed network connection Nov 30 14:06:44.864768752 (510h2m43.416239213s) [noise=true , address=145.79.227.22:4510 , piece_id=HZTR57BAQGRH64TGGKV3L62TVMHD2XOB7OQ7EOYZWYDI4HCZUUKQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52034->145.79.227.22:4510: use of closed network connection Nov 30 14:06:44.864954365 (510h2m43.416424826s) [noise=true , address=145.79.227.13:4803 , piece_id=TOP7RYT56WCL57JYDHFH3TCDAIB463PV574AN34QIP4Z2QZ4H4UA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56910->145.79.227.13:4803: use of closed network connection Nov 30 14:06:44.865000753 (510h2m43.416471214s) [noise=true , address=145.79.227.21:4473 , piece_id=VKC3KNRSYIUYILWXAERZZQ4WPZSRUKZDAMRHLSZBBK4LVWR444DQ, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60114->145.79.227.21:4473: use of closed network connection Nov 30 14:06:44.865045529 (510h2m43.41651598s) [noise=true , address=145.79.227.26:4655 , piece_id=3OZ3COXW53Z6VZXCIRVLIKNWRAWXNG7JW3H4TD4Q4CKFEQGEG3WA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37290->145.79.227.26:4655: use of closed network connection Nov 30 14:06:44.865124840 (510h2m43.416595311s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3GO3O7WKKRLMU5ZIST6WGFFLKDBXJYNF3URLGBAGGN2VTJQPOYA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32954->145.79.227.4:4186: use of closed network connection Nov 30 14:06:44.865152884 (510h2m43.416623335s) [noise=true , address=145.79.227.34:4907 , piece_id=5W43MRSPRBDPQ4VB4HBO4RN3G2BABVMUXF473QNHRQL4TPE4U5LA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56062->145.79.227.34:4907: use of closed network connection Nov 30 14:06:44.865222386 (510h2m43.416692847s) [noise=true , address=145.79.227.14:4837 , piece_id=HRVTRZMN26BOPKFLDEMGQGLWXUOMUFY7SEVTNTBJCGZUHCMHDSNA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37098->145.79.227.14:4837: use of closed network connection Nov 30 14:06:44.865282800 (510h2m43.416753251s) [noise=true , address=145.79.227.11:4732 , piece_id=RNEHYHFQGVCT5SQXI5S56KKYHUE4D5GODGHASR3JSDUEQLWCPKAA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36654->145.79.227.11:4732: use of closed network connection Nov 30 14:06:44.865535031 (510h2m43.417005482s) [noise=true , address=145.79.227.10:4402 , piece_id=FURED6JNT6HLPEI5H6AZWP6QSWVYLOZ2FLAMAF5QKKFNLOSBEDQA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:44888->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865639651 (510h2m43.417110112s) [noise=true , address=145.79.227.12:4765 , piece_id=N6DJ4TM22WVZO56FE76IQLIYS6W7QKL3JJY553HIHHTIC6FFDRKA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: read tcp 188.166.225.170:53870->145.79.227.12:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865769788 (510h2m43.417240249s) [noise=true , address=145.79.227.44:4193 , piece_id=IRC6ECVPRA2LQRTOX2DIRQJWTRZHLSJC7CR5UWM6H3UZDK6EXP4A, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: read tcp 188.166.225.170:33090->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865779507 (510h2m43.417249968s) [noise=true , address=145.79.227.18:4006 , piece_id=AL3BDZ6OXKXM2KJSBMA7IYQEXW725Y575PESGFMJLGZHLEJSE3SA, seg_pos={0 0}, upload=449157]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:34566->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:06:44.865853979 (510h2m43.41732443s) [seg_pos={0 0}, upload=449157]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Nov 30 14:06:45.115762223 (510h2m43.667232684s) [upload=449157]: Done uploading segments... Nov 30 14:06:45.115801749 (510h2m43.66727221s) [upload=449157]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.813002357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................X.......X........XX.X..X..............X...X] duration: 5.001153254s trace: https://jaeger.dev.storj.io/trace/338ed1a744a49667 FAIL (2025-11-30 14:29:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001177771s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)