Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 504970 reports (Page 5554 of 10100)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.724655439s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:34:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.724693781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.33029046s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:33:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.330315829s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.1241564s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:33:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.124199413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X........................X.........................] duration: 3.513567705s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:33:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513595058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X........................X.XXXXX..XX.X...........X.........] duration: 8.417259546s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:33:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.417288202s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.91281019s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.912834908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 6.004951856s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:33:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.004979378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.874350515s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:33:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.874383358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X........................X........................] duration: 3.471023908s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:32:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471061208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 7.454923904s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.454945295s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.313654491s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:32:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.313683127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.976674537s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:32:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.976707069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.309345784s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:32:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.309373756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X........................X.XXXXX..XX.X...........X........] duration: 9.773449382s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.773574972s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.96183615s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.961861237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.30501355s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.305044237s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.458995902s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.459027904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X........................X.XXXXX..XX.X...........X.......] duration: 7.282619706s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:32:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.282650425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 6.384551054s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:31:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.384571344s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.316508694s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.316537198s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X........................X.......................] duration: 3.769080576s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:31:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.769109862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.829804426s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:31:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.829834424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.236911936s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.236941553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X........................X.XXXXX..XX.X...........X......] duration: 9.012202583s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:31:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.012237448s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X........................X......................] duration: 3.68080511s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:30:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.680831639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.307450464s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:30:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.307550815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 6.503500781s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.503527281s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.277504313s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:30:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.277535873s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.118458568s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.118489699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X........................X.XXXXX..XX.X...........X.....] duration: 14.52825601s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:30:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.528281358s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X........................X.....................] duration: 3.784468109s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:30:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.78449451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.116640659s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:30:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.116669743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.311264038s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:30:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.311290179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 6.600476176s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:30:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.600512376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.942164443s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:30:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.942193278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.982906289s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:30:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.982932999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X........................X....................] duration: 3.910361176s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:29:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.910379742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.339764592s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:29:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.339791894s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.913763391s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:29:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.913789612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X........................X...................] duration: 3.573627117s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:29:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.573676613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X........................X.XXXXX..XX.X...........X....] duration: 9.714626597s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:29:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.714656204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.929614901s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:29:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.9296468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.868638614s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:28:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.868663812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.318890941s trace: https://jaeger.dev.storj.io/trace/20f1ef8c79d216e5 FAIL (2025-10-31 22:28:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.31891625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 7.754738894s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:28:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.754765985s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.197969841s trace: https://jaeger.dev.storj.io/trace/16457411f40d1a7a FAIL (2025-10-31 22:28:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.197993144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X........................X.XXXXX..XX.X...........X...] duration: 22.555581824s trace: https://jaeger.dev.storj.io/trace/5a7abee4f8025333 FAIL (2025-10-31 22:28:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.555607885s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X........................X..................] duration: 3.771728847s trace: https://jaeger.dev.storj.io/trace/150d6d8439e49137 FAIL (2025-10-31 22:28:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.771760978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 9.682248107s trace: https://jaeger.dev.storj.io/trace/777326af11170570 FAIL (2025-10-31 22:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:54:10.455403616 (34h13m42.151252369s) [upload=30026]: Starting upload Oct 31 21:54:10.455522532 (34h13m42.151371285s) [upload=30026]: Done starting upload Oct 31 21:54:10.455607515 (34h13m42.151456268s) [upload=30026]: Uploading segments... Oct 31 21:54:10.455903067 (34h13m42.15175182s) [upload=30026]: Got next segment. Inline: false Oct 31 21:54:10.455945959 (34h13m42.151794702s) [upload=30026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:10.455953403 (34h13m42.151802146s) [upload=30026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:10.703207265 (34h13m42.399056029s) [seg_pos={0 0}, upload=30026]: Begin upload segment... Oct 31 21:54:10.703270286 (34h13m42.399119039s) [seg_pos={0 0}, upload=30026]: Done begin upload segment. Oct 31 21:54:10.703295072 (34h13m42.399143825s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703311003 (34h13m42.399159756s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.703452613 (34h13m42.399301366s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704249731 (34h13m42.400098484s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704696551 (34h13m42.400545305s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.704999748 (34h13m42.400848531s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705356077 (34h13m42.40120483s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.705741331 (34h13m42.401590074s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706164356 (34h13m42.402013109s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706471972 (34h13m42.402320715s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.706819254 (34h13m42.402667997s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707176183 (34h13m42.403024936s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.707611081 (34h13m42.403459834s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708027884 (34h13m42.403876637s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708460038 (34h13m42.404308781s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:10.708841935 (34h13m42.404690688s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Uploading piece... Oct 31 21:54:12.455521019 (34h13m44.151369772s) [upload=30026]: Next returned nil segment Oct 31 21:54:12.455539354 (34h13m44.151388107s) [upload=30026]: Waiting for error group managing segments... Oct 31 21:54:12.740273646 (34h13m44.436122389s) [noise=true , address=145.79.227.14:4837 , piece_id=L4U4WTLR3JG7HM3MGRXLBTAFDDP3ZBD72FJWZPHAFQKMS2IEVM4A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742133407 (34h13m44.43798216s) [noise=true , address=145.79.227.8:4331 , piece_id=AXFWSHM2G7TPVCVWUOCBI2WM2KY5DLAOMS3S4W3VD5GW2347BZJQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.742843700 (34h13m44.438692443s) [noise=true , address=145.79.227.30:4763 , piece_id=TRG64EI7L2TPOJLZMNBYHGGGQNPHQ7SJLQE5MTF6JCLVURCPBHQQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.743356016 (34h13m44.439204769s) [noise=true , address=145.79.227.18:4005 , piece_id=QDIXJD4MHGQ54G52GVGUUTWKZ44O6CPZL6U2564FG7LGY4GEV52A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.747176791 (34h13m44.443025535s) [noise=true , address=145.79.227.11:4730 , piece_id=QFZX4JWD6FWSONWUGIHRJJXHFX7OVHI3AMV5A2FGVWWWQ2NWXYSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.749287941 (34h13m44.445136684s) [noise=true , address=145.79.227.13:4802 , piece_id=E7JK55QB55MHBEYQX3TFOUKEZINXIA3BVWXVCBHB6SPCQG57GUHQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.751847415 (34h13m44.447696168s) [noise=true , address=145.79.227.24:4583 , piece_id=VGCLUWE3VD57GRMBFXGOWUCNISPGKSZMA5Z4YNE347SSO5RG4U7A, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.754964321 (34h13m44.450813084s) [noise=true , address=145.79.227.17:4971 , piece_id=KRYYUBN3ZG5UOHAJW4GDBRHMPZI3LHD6VIAIV3TEO44UZ4RRWPHA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.760193528 (34h13m44.456042281s) [noise=true , address=145.79.227.28:4727 , piece_id=CBLLSRNFJSLZY4JS6BMMFASRHGMPYWBRZJTYFDLLUMHGS43YGRNA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.763852014 (34h13m44.459700757s) [noise=true , address=145.79.227.5:4223 , piece_id=V74DYYH2ZR63E36KO3CQHRW6WHKHYRZSZCUE74YV5U47LGMMTECQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:12.804034039 (34h13m44.499882792s) [noise=true , address=145.79.227.13:4804 , piece_id=O2E5VDS3UKDNZAP57AEMCBBIX2FZ34RMTEUXIFHEO6HOXKBEE3TQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: <nil> Oct 31 21:54:39.766396977 (34h14m11.46224573s) [noise=true , address=77.90.31.19:4841 , piece_id=73OQRYSAIH4ATTLZUU2XPPX7LTUZU4BALKSTIKKFJI26UHMOMZYQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37328->77.90.31.19:4841: use of closed network connection Oct 31 21:54:39.766509131 (34h14m11.462357874s) [noise=true , address=77.90.31.18:4807 , piece_id=ACQ25HNYKBAZWIZXQI5GAIV6JF4DKB2KDNDI3Y2BMPX2BOSL4CUA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:45860->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766523238 (34h14m11.462371981s) [noise=true , address=77.90.31.13:4628 , piece_id=OEUUWGKPZRWXUA2QNKA4WDJH72IQJWTEW2EEVBRBCMJJIMO5TRWQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: read tcp 188.166.225.170:49310->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766612227 (34h14m11.46246097s) [noise=true , address=45.91.116.14:4190 , piece_id=W6O2FOFZVCJLNSCRITL76RWMFXMU3ABTQOAQY5BBANHPVGARVEBQ, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:52980->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766670488 (34h14m11.462519241s) [noise=true , address=45.91.116.4:4939 , piece_id=DHPRQICX65O45C3ZDINAHGJYBI64LH375KXYQWCWDFPSPYSUVZSA, seg_pos={0 0}, upload=30026]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:49096->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:39.766695265 (34h14m11.462544008s) [seg_pos={0 0}, upload=30026]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:54:40.008340748 (34h14m11.704189501s) [upload=30026]: Done uploading segments... Oct 31 21:54:40.008374502 (34h14m11.704223246s) [upload=30026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.682278023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.355284822s trace: https://jaeger.dev.storj.io/trace/4a436e8f0e3652a9 FAIL (2025-10-31 22:28:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841512922 (34h13m43.537361665s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.841695811 (34h13m43.537544554s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842121341 (34h13m43.537970084s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842258863 (34h13m43.538107606s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.842653173 (34h13m43.538501917s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843123740 (34h13m43.538972493s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843233278 (34h13m43.539082031s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.843685530 (34h13m43.539534283s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844090440 (34h13m43.539939193s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844193336 (34h13m43.540042089s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844586345 (34h13m43.540435098s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.844901665 (34h13m43.540750418s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845319241 (34h13m43.541167994s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845683333 (34h13m43.541532086s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.845754208 (34h13m43.541602961s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:11.846188546 (34h13m43.542037289s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Uploading piece... Oct 31 21:54:13.235177054 (34h13m44.931025807s) [upload=30028]: Next returned nil segment Oct 31 21:54:13.235196149 (34h13m44.931044902s) [upload=30028]: Waiting for error group managing segments... Oct 31 21:54:13.731846523 (34h13m45.427695276s) [noise=true , address=145.79.227.12:4772 , piece_id=UGY766HRNYHSEJ7ZI42GUCQRVQWD5KPPHIZOPIKEW6KYR6HDZG5Q, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.746649450 (34h13m45.442498203s) [noise=true , address=145.79.227.14:4842 , piece_id=KW77HQOPLMDF2KRNAB2YQJP5TPGVDW373TTQZ65DEFSWQJYBISQA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.749172355 (34h13m45.445021108s) [noise=true , address=145.79.227.42:4128 , piece_id=KUUKI4ITT2NHQCBRESJCNVTL27QKHPMUHJEVZHBTTTHI7FHQ2RQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.824326073 (34h13m45.520174826s) [noise=true , address=145.79.227.2:4124 , piece_id=5N45IAC7BXFDSV47RDZQMHRYXGREYE5H6YTYXDROYQSQABBOFANA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.838257411 (34h13m45.534106154s) [noise=true , address=145.79.227.29:4974 , piece_id=UI7BBVOT65B3HO52C3AVESWQYPST43E3CBU4IB5HNZAFVHGJ2F2A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:13.846435784 (34h13m45.542284527s) [noise=true , address=145.79.227.21:4498 , piece_id=PES7TMXZSENFZP5IMOZGDPXWCKQUBC7GAEOEBRNZUXVPZYDIPBJA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:14.712603264 (34h13m46.408452007s) [noise=true , address=145.79.227.16:4925 , piece_id=6SC5672OVXERGWTKXF5VV3PLHCBTJK36E6XRLLZMDE4PSNOTMUQQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:45.241231995 (34h14m16.937080748s) [noise=true , address=77.90.31.16:4743 , piece_id=IFRBYYMGBJVI42ZDQ6DOE4AF3FSTWVYW4NOOJLXR3HLHQRXG2F6A, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:47.222044337 (34h14m18.91789309s) [noise=true , address=77.90.31.13:4647 , piece_id=RNEAPD5I7YUI4OFZT6KPTW4TYKIW4GD5D5FXDHRHAGQGOM37CWLQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:54:57.928194315 (34h14m29.624043068s) [noise=true , address=77.90.31.15:4719 , piece_id=QGBNDSP6TQTFLUWVPVB5VEW2VMRBZTJRLGAVSMAD5S2K5HKRRCWQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:07.960970275 (34h14m39.656819028s) [noise=true , address=77.90.31.11:4560 , piece_id=2SMJVHCSKCB766N55TK7PIJNOSHKEUXQRLKT77RQ5GFUEW2DI5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: <nil> Oct 31 21:55:10.903651570 (34h14m42.599500323s) [noise=true , address=45.91.116.10:4106 , piece_id=KLB7VKSEJPVZ3XUNK3QKOKGWJK6SJKLNGIUJLDOALQEIZLUQUFSA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ, address:45.91.116.10:4106): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.903817216 (34h14m42.599665969s) [noise=true , address=45.91.116.9:4277 , piece_id=3DDI5XLPDI7J7X5V6TXTQCJJGYBUD6NDK5ZUWMSAJWBRVMFZ34ZQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56658->45.91.116.9:4277: use of closed network connection Oct 31 21:55:10.903935131 (34h14m42.599783884s) [noise=true , address=45.91.116.11:4140 , piece_id=OLY2BJMBLENIFP2SQYY6G6V5MIDN7HKJWEU4ZEBQ7AQRDMFMNKRA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35734->45.91.116.11:4140: use of closed network connection Oct 31 21:55:10.904041113 (34h14m42.599889866s) [noise=true , address=45.91.116.14:4199 , piece_id=ZWJUY4JDZFC3PPNUE6MM7ZAFD3ZJ6QH4SZCZMNEQTKH5SR4OS7GQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47748->45.91.116.14:4199: use of closed network connection Oct 31 21:55:10.904246463 (34h14m42.600095216s) [noise=true , address=77.90.31.14:4674 , piece_id=62IZARJIWZYWAKAVHOP32MZUSTE35VBMA46K472HY44SFP63G5EA, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, address:77.90.31.14:4674): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904369097 (34h14m42.60021785s) [noise=true , address=45.91.116.7:4044 , piece_id=QKGXW4726NTBO3R6X3AZM2JHQ3DS7O3GA33U2DI36RG7H2RM74CQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35472->45.91.116.7:4044: use of closed network connection Oct 31 21:55:10.904515886 (34h14m42.600364639s) [noise=true , address=77.90.31.17:4787 , piece_id=X3PYJKDFMSJCMGAKGIL6JEILLOWA2MV2TDYLP3Y7QGTWFOYALZRQ, seg_pos={0 0}, upload=30028]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:55:10.904544992 (34h14m42.600393735s) [seg_pos={0 0}, upload=30028]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:55:11.152804097 (34h14m42.84865284s) [upload=30028]: Done uploading segments... Oct 31 21:55:11.152854594 (34h14m42.848703337s) [upload=30028]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.355337392s core uplink us-select-1 us1-placement-20

1
...
5552
5553
5554
5555
5556
...
10100