Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.961434778s trace: https://jaeger.dev.storj.io/trace/36fe49f5e04f5b64 FAIL (2025-11-30 15:23:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 4.961463544s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X...XX..............X.........X.......X................X...] duration: 3.513323924s trace: https://jaeger.dev.storj.io/trace/14b4299b99e1723a FAIL (2025-11-30 15:23:13 UTC) orbiter warning: context deadline exceeded Duration: 3.513354892s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 12.114879983s trace: https://jaeger.dev.storj.io/trace/2b6602aaba388de5 FAIL (2025-11-30 15:22:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.114910741s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X...XX..............X.........X.......X................X..] duration: 3.491016504s trace: https://jaeger.dev.storj.io/trace/14b4299b99e1723a FAIL (2025-11-30 15:22:48 UTC) orbiter warning: context deadline exceeded Duration: 3.491047131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.227612355s trace: https://jaeger.dev.storj.io/trace/36fe49f5e04f5b64 FAIL (2025-11-30 15:22:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 3.227641441s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X...XX..............X.........X.......X................X.] duration: 3.673690907s trace: https://jaeger.dev.storj.io/trace/14b4299b99e1723a FAIL (2025-11-30 15:22:29 UTC) orbiter warning: context deadline exceeded Duration: 3.673722136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 16.31983314s trace: https://jaeger.dev.storj.io/trace/2b6602aaba388de5 FAIL (2025-11-30 15:22:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.319858308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.767010399s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.767040907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X...XX..............X.........X.......X................X] duration: 5.002059514s trace: https://jaeger.dev.storj.io/trace/14b4299b99e1723a FAIL (2025-11-30 15:22:07 UTC) orbiter warning: context deadline exceeded Duration: 5.002089761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.076403081s trace: https://jaeger.dev.storj.io/trace/36fe49f5e04f5b64 FAIL (2025-11-30 15:21:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.076442435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.783330256s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:21:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.783360495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X...XX..............X.........X.......X................] duration: 3.315367034s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:21:46 UTC) orbiter warning: context deadline exceeded Duration: 3.3153966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 11.063991646s trace: https://jaeger.dev.storj.io/trace/2b6602aaba388de5 FAIL (2025-11-30 15:21:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.064017445s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X...XX..............X.........X.......X...............] duration: 3.536098331s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:21:24 UTC) orbiter warning: context deadline exceeded Duration: 3.536125984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002346767s trace: https://jaeger.dev.storj.io/trace/2b6602aaba388de5 FAIL (2025-11-30 15:21:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002407665s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.806799735s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:20:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.806829562s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 10.547717093s trace: https://jaeger.dev.storj.io/trace/36fe49f5e04f5b64 FAIL (2025-11-30 15:20:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.547755136s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X...XX..............X.........X.......X..............] duration: 3.715365725s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:20:31 UTC) orbiter warning: context deadline exceeded Duration: 3.715394779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.798437631s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:19:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.798465215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X...XX..............X.........X.......X.............] duration: 3.746805264s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:19:37 UTC) orbiter warning: context deadline exceeded Duration: 3.746835169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.086496511s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:19:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.086527621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X...XX..............X.........X.......X............] duration: 3.452525549s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:18:26 UTC) orbiter warning: context deadline exceeded Duration: 3.452550346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.769107134s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.769128083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X...XX..............X.........X.......X...........] duration: 3.749819922s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:17:26 UTC) orbiter warning: context deadline exceeded Duration: 3.749847635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.793996754s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:17:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.794020951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X...XX..............X.........X.......X..........] duration: 3.749100213s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:16:47 UTC) orbiter warning: context deadline exceeded Duration: 3.749190625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X...XX..............X.........X.......X.........] duration: 4.406251456s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:16:32 UTC) orbiter warning: context deadline exceeded Duration: 4.406278468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.844701479s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.844728741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.829911255s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.829943376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X...XX..............X.........X.......X........] duration: 3.648684765s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:15:28 UTC) orbiter warning: context deadline exceeded Duration: 3.648712788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.............X...XX..............X.........X.......X.......] duration: 3.683318603s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:14:26 UTC) orbiter warning: context deadline exceeded Duration: 3.683343319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.748839908s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.748873782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.............X...XX..............X.........X.......X......] duration: 3.523524212s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:13:34 UTC) orbiter warning: context deadline exceeded Duration: 3.523553957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.849901089s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:13:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.849960884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.811452457s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:13:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.811494076s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.............X...XX..............X.........X.......X.....] duration: 3.521600778s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:12:44 UTC) orbiter warning: context deadline exceeded Duration: 3.521633332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.766574868s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.766604303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.849530705s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:11:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.849560621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.............X...XX..............X.........X.......X....] duration: 3.358069971s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:11:47 UTC) orbiter warning: context deadline exceeded Duration: 3.358100758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.815398007s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:11:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.815427713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X...X.............X...XX..............X.........X.......X...] duration: 3.46971481s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:11:16 UTC) orbiter warning: context deadline exceeded Duration: 3.469743122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.761589002s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:10:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.761616925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X...X.............X...XX..............X.........X.......X..] duration: 3.563931202s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:10:27 UTC) orbiter warning: context deadline exceeded Duration: 3.563966549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.069195168s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.06922762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X...X.............X...XX..............X.........X.......X.] duration: 3.705004497s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:09:13 UTC) orbiter warning: context deadline exceeded Duration: 3.705031008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.135509848s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:09:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.135544434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.962470685s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:08:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.962501333s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X...X.............X...XX..............X.........X.......X] duration: 5.001665122s trace: https://jaeger.dev.storj.io/trace/5c79dc2c9bf52875 FAIL (2025-11-30 15:08:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001689577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.889078279s trace: https://jaeger.dev.storj.io/trace/605d8a63fb96e8df FAIL (2025-11-30 15:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.755770725 (510h34m32.307241176s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:33.758005552 (510h34m32.309476013s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Uploading piece... Nov 30 14:38:35.558744798 (510h34m34.110215269s) [upload=449630]: Next returned nil segment Nov 30 14:38:35.558762652 (510h34m34.110233113s) [upload=449630]: Waiting for error group managing segments... Nov 30 14:38:35.854088201 (510h34m34.405558662s) [noise=true , address=45.91.116.4:4938 , piece_id=PLM35KHFFYVLD4GJP2ASUNJDWJBTHJ2KW6TZIFDIVJOJHVHUTD7A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:38:35.908542693 (510h34m34.460013154s) [noise=true , address=77.90.31.19:4842 , piece_id=Y3RNG5SGFVZUSGQL3RVG7MXZ3YLY5LOFTPGQJE4V6WIFMPXQVOAA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: <nil> Nov 30 14:39:02.804142302 (510h35m1.355612783s) [noise=true , address=145.79.227.38:4979 , piece_id=ITRTQMCJB6EYO5DEUEJFHCT4TKSA2QDFHSDQV47ENBWQMUO6TU3A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41868->145.79.227.38:4979: use of closed network connection Nov 30 14:39:02.804344558 (510h35m1.355815029s) [noise=true , address=145.79.227.28:4726 , piece_id=NWCHNYTKDVNPWCTPUG35ATMTN7DQRQJBMAQHA4KKHIY2BPZ6QTEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43432->145.79.227.28:4726: use of closed network connection Nov 30 14:39:02.804505384 (510h35m1.355975845s) [noise=true , address=145.79.227.29:4941 , piece_id=I6EEMSPAWAB3IFLXQQ2XJULBMAUMOAC3EMWN2HUI5QHVRBT37JEQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:145.79.227.29:4941): protocol: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57740->145.79.227.29:4941: use of closed network connection Nov 30 14:39:02.804657604 (510h35m1.356128065s) [noise=true , address=145.79.227.6:4257 , piece_id=QRBO3ULY5TQ6NAUMCN7AFKRZ7AA6AUIZDBUY6C7CSDUXQ5H3DPUA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52358->145.79.227.6:4257: use of closed network connection Nov 30 14:39:02.804824892 (510h35m1.356295354s) [noise=true , address=145.79.227.39:4013 , piece_id=O7PCKE4LSLQ2YMJWX2MD7V5OUE2F7LGESM7ZBBKU6VPGRXPIEDRQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36784->145.79.227.39:4013: use of closed network connection Nov 30 14:39:02.804979296 (510h35m1.356449757s) [noise=true , address=145.79.227.26:4653 , piece_id=COZBZBNJ2W2YXNHHL3YBOFZWKF6MD3X27RFZ35MEQDFUOMAA3UNQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41138->145.79.227.26:4653: use of closed network connection Nov 30 14:39:02.805156744 (510h35m1.356627205s) [noise=true , address=145.79.227.24:4581 , piece_id=DAVNMZGXLR4WFRPBHQK2IU6JLPTQFUAVQD6LMHDYV7K4OGQOAG3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48786->145.79.227.24:4581: use of closed network connection Nov 30 14:39:02.805341927 (510h35m1.356812418s) [noise=true , address=145.79.227.27:4689 , piece_id=CSOF346QAPCHSSTO4R5BXT5QFZWH775QDJ7EH2YUXVQV2QDB22GQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60622->145.79.227.27:4689: use of closed network connection Nov 30 14:39:02.805528983 (510h35m1.356999444s) [noise=true , address=145.79.227.2:4115 , piece_id=7MQZJM5TJ67JUN4F7FLONSMHRB5UULA7GLBEROCGDTS3VNLS4Q6A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50116->145.79.227.2:4115: use of closed network connection Nov 30 14:39:02.806228735 (510h35m1.357699206s) [noise=true , address=145.79.227.40:4050 , piece_id=JSF6LEWDSWCAEE6252TS5JBUDO26BVB6PVMRSHXGK6I3FCHFF2LQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.806549115 (510h35m1.358019586s) [noise=true , address=145.79.227.5:4222 , piece_id=WA4IAQLFWVY5IRSB34OYJCQKV2RMVTQWXWKCYS4OZSDAWAWSKL3Q, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42860->145.79.227.5:4222: use of closed network connection Nov 30 14:39:02.806849267 (510h35m1.358319728s) [noise=true , address=145.79.227.8:4329 , piece_id=ST7FVM7SCOZYHA7TKBWWIZNVJM6AHUQAB5CUWST3MSVTH5AS3ERA, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46744->145.79.227.8:4329: use of closed network connection Nov 30 14:39:02.807262093 (510h35m1.358732584s) [noise=true , address=145.79.227.3:4149 , piece_id=A24R3XEJTS4GCIEEY7NNDBAFRTWLV3AVSZXF5C6UONFZFG6IH75A, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47520->145.79.227.3:4149: use of closed network connection Nov 30 14:39:02.809083602 (510h35m1.360554094s) [noise=true , address=145.79.227.17:4972 , piece_id=SNSBY3DCFZE3YYU3LATMSRAEBIINJSA4VGCQANPLUMRROLVRMXUQ, seg_pos={0 0}, upload=449630]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:52300->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 14:39:02.809192580 (510h35m1.360663041s) [seg_pos={0 0}, upload=449630]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Nov 30 14:39:03.056238374 (510h35m1.607708835s) [upload=449630]: Done uploading segments... Nov 30 14:39:03.056278069 (510h35m1.60774853s) [upload=449630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.889106864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X...X.............X...XX..............X.........X.......] duration: 3.815614809s trace: https://jaeger.dev.storj.io/trace/4934c22573536b0b FAIL (2025-11-30 15:07:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815661097s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)