Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 504963 reports (Page 5553 of 10100)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 309.629108ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:39:30 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 309.659054ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X........................X...............................] duration: 3.478686973s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:39:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478726478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.681088066s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:39:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.681115647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X.XXXXX..XX.X...........X...............] duration: 6.894026558s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.894050515s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.332497403s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:38:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.332520897s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 319.196165ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:38:31 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 319.210031ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.792116407s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:38:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.792137348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X........................X..............................] duration: 3.535010648s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:38:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535044181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.325509177s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:38:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325536137s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................X.XXXXX..XX.X...........X..............] duration: 10.409639426s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:37:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.409674372s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 559.606165ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:37:44 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 559.634789ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.402072477s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:37:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.402100059s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.038022551s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.038049692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X........................X.............................] duration: 3.495064349s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:37:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495095548s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.530591314s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:36:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.530617293s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 296.867016ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:36:52 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 296.892485ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................X.XXXXX..XX.X...........X.............] duration: 6.997362259s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:36:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.997397256s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.331995356s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.332022728s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X........................X............................] duration: 3.431359584s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:36:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431400001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 6.197230699s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:36:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.197259092s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 333.83401ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:36:22 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 333.864096ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................X.XXXXX..XX.X...........X............] duration: 11.52321963s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:36:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.523247012s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.82160583s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:36:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.821659483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.328547724s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:36:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.328580716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 6.236312786s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.236341682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.823174487s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:36:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.823199033s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 311.076875ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:35:52 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 311.099026ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 6.089822332s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:35:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.089878439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X........................X...........................] duration: 3.50078862s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:35:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500847752s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.723921929s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:35:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723966233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.313643961s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:35:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.313673787s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.933032016s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:35:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.933064429s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 343.946285ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:35:19 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 343.987894ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................X.XXXXX..XX.X...........X...........] duration: 8.258431405s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:35:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.258477413s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 6.307795005s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.307824409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.00379777s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:34:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.003827056s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 418.88991ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/572052ad3b94b02 FAIL (2025-10-31 22:34:36 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:60884->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 418.91633ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X........................X..........................] duration: 3.484396385s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:34:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484426101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 6.716330761s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:34:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.716358273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X........................X.XXXXX..XX.X...........X..........] duration: 11.137279961s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:34:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.137313164s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.790151576s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:34:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.790177886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.357710725s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:34:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.357729671s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.213560623s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:34:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.213587645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.724655439s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:34:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.724693781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.33029046s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:33:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.330315829s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.1241564s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:33:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.124199413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X........................X.........................] duration: 3.513567705s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:33:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513595058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X........................X.XXXXX..XX.X...........X.........] duration: 8.417259546s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:33:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.417288202s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.91281019s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.912834908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 6.004951856s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:33:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.004979378s core uplink us-select-1

1
...
5551
5552
5553
5554
5555
...
10100