Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.3241543s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:10:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.324175472s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 11.737863266s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:10:51 UTC) orbiter warning: context deadline exceeded Duration: 11.737893042s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 14.906623021s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:10:22 UTC) orbiter warning: context deadline exceeded Duration: 14.906651946s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XX.X....XXXX...X.XXX.XXXX..................................] duration: 3.660534584s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:10:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.660565012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.934504256s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:10:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.934530124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X.XX.X....XXXX...X.XXX.XXXX.................................] duration: 3.5429465s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:09:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542975626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 13.474863151s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:09:31 UTC) orbiter warning: context deadline exceeded Duration: 13.474880324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.252342033s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.252387308s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XX.XX.X....XXXX...X.XXX.XXXX................................] duration: 3.519719739s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:08:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519743243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 7.091223994s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:08:47 UTC) orbiter warning: context deadline exceeded Duration: 7.09126991s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.031812661s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:08:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.031842578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.XX.XX.X....XXXX...X.XXX.XXXX...............................] duration: 3.548714451s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:08:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548833658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 12.025454934s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:08:12 UTC) orbiter warning: context deadline exceeded Duration: 12.025476464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.006346104s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.006375601s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 6.161734562s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:07:44 UTC) orbiter warning: context deadline exceeded Duration: 6.161760954s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.XX.XX.X....XXXX...X.XXX.XXXX..............................] duration: 3.569151352s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:07:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569175558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X.XX.XX.X....XXXX...X.XXX.XXXX.............................] duration: 3.571302157s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:06:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571331393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 11.713917466s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:06:42 UTC) orbiter warning: context deadline exceeded Duration: 11.713946802s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X.XX.XX.X....XXXX...X.XXX.XXXX............................] duration: 3.482670764s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:06:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482748462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.469432316s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:06:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.469461822s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 12.315363854s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:06:04 UTC) orbiter warning: context deadline exceeded Duration: 12.315393952s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X.XX.XX.X....XXXX...X.XXX.XXXX...........................] duration: 3.620532737s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:05:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.620586611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.139163174s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:05:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.13919314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.416298227s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:05:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.416324497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X...X.XX.XX.X....XXXX...X.XXX.XXXX..........................] duration: 3.504854806s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:05:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504882999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 12.716012186s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:05:09 UTC) orbiter warning: context deadline exceeded Duration: 12.716047903s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X...X.XX.XX.X....XXXX...X.XXX.XXXX.........................] duration: 3.570959324s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:04:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571011324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 12.863377278s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:04:44 UTC) orbiter warning: context deadline exceeded Duration: 12.863408528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.061357967s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:04:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.061390619s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..X...X.XX.XX.X....XXXX...X.XXX.XXXX........................] duration: 3.676275805s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:03:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.676307196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 13.341969729s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:03:35 UTC) orbiter warning: context deadline exceeded Duration: 13.342003332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.239237907s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:03:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.239278663s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X...X.XX.XX.X....XXXX...X.XXX.XXXX.......................] duration: 3.497073076s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:02:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497101661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 13.370323532s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:02:22 UTC) orbiter warning: context deadline exceeded Duration: 13.37034867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.354217062s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:02:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.354263691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....X...X.XX.XX.X....XXXX...X.XXX.XXXX......................] duration: 3.570362668s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:02:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570388838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 16.717560806s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:01:41 UTC) orbiter warning: context deadline exceeded Duration: 16.717781095s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.....X...X.XX.XX.X....XXXX...X.XXX.XXXX.....................] duration: 3.582635009s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582657852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.289402786s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:01:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.289431341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.043433027s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:00:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.043449479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [......X...X.XX.XX.X....XXXX...X.XXX.XXXX....................] duration: 3.641905134s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:00:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.641925916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 13.919012422s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:00:39 UTC) orbiter warning: context deadline exceeded Duration: 13.91904238s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.157659182s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:00:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.157692446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.......X...X.XX.XX.X....XXXX...X.XXX.XXXX...................] duration: 3.604929563s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:00:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604956775s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 13.474166945s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 14:59:46 UTC) orbiter warning: context deadline exceeded Duration: 13.474241767s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.147642299s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 14:59:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.147776145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [........X...X.XX.XX.X....XXXX...X.XXX.XXXX..................] duration: 3.482027757s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 14:59:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482054157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.136253821s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 14:59:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.13627376s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 12.401304473s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 14:59:00 UTC) orbiter warning: context deadline exceeded Duration: 12.401335992s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X...X.XX.XX.X....XXXX...X.XXX.XXXX.................] duration: 3.428874057s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 14:58:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42889573s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)