Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.093733907s FAIL (2025-09-27 05:01:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.093765678s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.432836019s FAIL (2025-09-27 05:00:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.432864563s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.332716185s FAIL (2025-09-27 04:59:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.332746152s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.216697747s FAIL (2025-09-27 04:59:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.216718266s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.682861109s FAIL (2025-09-27 04:58:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.682890914s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000279287s FAIL (2025-09-27 04:58:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 30.000309524s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.977995016s FAIL (2025-09-27 01:31:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.978025904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.261952881s FAIL (2025-09-27 01:30:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.261995352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.023673976s FAIL (2025-09-27 01:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.023705347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.940722412s FAIL (2025-09-27 01:28:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940756248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.141030949s FAIL (2025-09-27 01:27:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.141065546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.173033349s FAIL (2025-09-27 01:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.173064059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.14499606s FAIL (2025-09-27 01:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.145081853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.983773359s FAIL (2025-09-27 01:25:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.983802352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.948634368s FAIL (2025-09-27 01:24:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.948670336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.056729428s FAIL (2025-09-27 01:23:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.056762409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.002273064s FAIL (2025-09-27 01:23:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.002302309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.144099714s FAIL (2025-09-27 01:22:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.14412914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.117527406s FAIL (2025-09-27 01:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.117567652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.949039217s FAIL (2025-09-27 01:20:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.94907713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.90183856s FAIL (2025-09-27 01:19:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.901895527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.138612478s FAIL (2025-09-27 01:18:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.138643986s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.976982406s FAIL (2025-09-27 01:17:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.977009619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.909787848s FAIL (2025-09-27 01:17:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909823105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.253359762s FAIL (2025-09-27 01:16:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.25339031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.415808519s FAIL (2025-09-27 01:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.415848454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.039256614s FAIL (2025-09-27 01:13:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.039289085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.971023751s FAIL (2025-09-27 01:13:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.971059279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.098777483s FAIL (2025-09-27 01:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.098846783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.107249347s FAIL (2025-09-27 01:11:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.107288251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.02935315s FAIL (2025-09-27 01:10:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.029402454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.154529962s FAIL (2025-09-27 01:10:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.154571912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.018712328s FAIL (2025-09-27 01:09:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.018738708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.054805613s FAIL (2025-09-27 01:08:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.054836963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.124349072s FAIL (2025-09-27 01:07:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.124378478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.096659066s FAIL (2025-09-27 01:06:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.096738737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.077087798s FAIL (2025-09-27 01:05:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.077118325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.118544522s FAIL (2025-09-27 01:05:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.11858552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.069864678s FAIL (2025-09-27 01:04:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.06989753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.115126809s FAIL (2025-09-27 01:03:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.11515925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.084534489s FAIL (2025-09-27 01:03:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.084571621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.057550286s FAIL (2025-09-27 01:02:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.057582297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.994644937s FAIL (2025-09-27 01:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.994671377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.043703339s FAIL (2025-09-27 01:00:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.04373477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.017731413s FAIL (2025-09-27 01:00:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.017775146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.207663684s FAIL (2025-09-27 00:59:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.207701278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.985529589s FAIL (2025-09-27 00:58:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.98556129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.913508853s FAIL (2025-09-27 00:57:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.913540424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.12451693s FAIL (2025-09-27 00:56:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.124548571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.919958168s FAIL (2025-09-27 00:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 00:44:21.707148230 (83h13m7.169240713s) [upload=73230]: Starting upload Sep 27 00:44:21.707246958 (83h13m7.169339441s) [upload=73230]: Done starting upload Sep 27 00:44:21.707505240 (83h13m7.169597723s) [upload=73230]: Uploading segments... Sep 27 00:44:21.707663371 (83h13m7.169755854s) [upload=73230]: Got next segment. Inline: false Sep 27 00:44:21.707703297 (83h13m7.16979578s) [upload=73230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 00:44:21.707708787 (83h13m7.16980127s) [upload=73230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 00:44:21.959972247 (83h13m7.42206474s) [seg_pos={0 0}, upload=73230]: Begin upload segment... Sep 27 00:44:21.960043873 (83h13m7.422136356s) [seg_pos={0 0}, upload=73230]: Done begin upload segment. Sep 27 00:44:21.960066537 (83h13m7.42215902s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960076606 (83h13m7.422169089s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.960137251 (83h13m7.422229734s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.961132437 (83h13m7.42322496s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962299769 (83h13m7.424392252s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.962958493 (83h13m7.425050966s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.963522026 (83h13m7.425614509s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.964077985 (83h13m7.426170458s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966296219 (83h13m7.428388702s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.966345362 (83h13m7.428437845s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968832257 (83h13m7.43092474s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.968914905 (83h13m7.431007388s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.971779149 (83h13m7.433871622s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.977396715 (83h13m7.439489199s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.982395014 (83h13m7.444487497s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:21.986687846 (83h13m7.44878033s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Uploading piece... Sep 27 00:44:23.581851509 (83h13m9.043943992s) [upload=73230]: Next returned nil segment Sep 27 00:44:23.581880514 (83h13m9.043972987s) [upload=73230]: Waiting for error group managing segments... Sep 27 00:44:23.962967780 (83h13m9.425060253s) [noise=true , address=145.79.227.3:4150 , piece_id=AI2QMRQJEYHX4YK3FE7LI7RCQSFFKTI5VNE43TSFO7YNSFBMLDKQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.963044627 (83h13m9.42513711s) [noise=true , address=145.79.227.19:4078 , piece_id=QK6SWMBKWY7LOFMLDJZTUQFSI65ROAJWBRGY2743GP5NVK7AIW3Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.972980585 (83h13m9.435073068s) [noise=true , address=145.79.227.15:4873 , piece_id=FFP7F6HRXXYBTFGIAPQBZV2T6AI6JSQIQQBJOVH4RXKHMWKWT3KQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.979740547 (83h13m9.44183303s) [noise=true , address=145.79.227.5:4221 , piece_id=KF5YC3LUNMNXNW27MRC25FWNYQUTUUIDE3CVR4XABW6HAMCEJ37Q, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985638287 (83h13m9.44773077s) [noise=true , address=145.79.227.16:4911 , piece_id=UQYH4DGJLJMCBDUS2LJD62KEB65TAHH252YQ2YTPREFUZDHI3BFA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.985769697 (83h13m9.447862281s) [noise=true , address=145.79.227.18:4006 , piece_id=EDZVZ2F4E6R6QI2L6YQVS2QJPEZYETJSAOK43C4E3O33S5FEAOBQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:23.997586448 (83h13m9.459678921s) [noise=true , address=145.79.227.11:4732 , piece_id=AX63NUUVNPQBYKXGSZIKN7E4ES53SHLDYEJHFD6CT3H6PBUYMGUA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073295487 (83h13m9.53538796s) [noise=true , address=45.91.116.11:4117 , piece_id=IC33US7RSNDGWX7OSF54IN7NSAIVO3ZMZULKIGODC3AAQSRZCXJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.073672045 (83h13m9.535764528s) [noise=true , address=45.91.116.9:4262 , piece_id=JQZEEVHRCZN5LQUPH3DBUEKZH2WPJ5MJIQ67EHBJEJTHBWJQK4TA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.098073267 (83h13m9.56016576s) [noise=true , address=45.91.116.5:4299 , piece_id=R3QFDHOU7SKSMMD6NFWDSLURZMCUTFNPVF3TKXGIEWCMTJBI5T4A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.099029988 (83h13m9.561122461s) [noise=true , address=45.91.116.11:4119 , piece_id=QJBF6B6AI6PQSAJDZPPXPARUYBHIQMRZME2DQFCC2BVPEZDMDRYA, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:24.170422869 (83h13m9.632515352s) [noise=true , address=45.91.116.10:4081 , piece_id=7JPLCBRR3S4KML2HCVTQPUTZ6OA5QQOZLBWWFP4WHEVZ3P3ILKJQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: <nil> Sep 27 00:44:50.959550021 (83h13m36.421642504s) [noise=true , address=77.90.31.15:4699 , piece_id=CCR4LWBFUCU3LBKCX2XU2RBSCUJYBT5GQWZSKH5P4TYG257P2HYQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51580->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 00:44:50.959671443 (83h13m36.421763926s) [noise=true , address=77.90.31.17:4771 , piece_id=DFL7GXQ4DE3GK7CYNGCFC4FHUKXOKPXCBN6KUV4PLFVNZPUA3QOQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959692773 (83h13m36.421785256s) [noise=true , address=77.90.31.10:4518 , piece_id=HKB4UVXHV32GYSOG2MNCVY3RUSV2XIWPQMYIFIXOW3LZ7AMJ345A, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959852898 (83h13m36.421945381s) [noise=true , address=77.90.31.16:4734 , piece_id=VSKAOODXJZNHJ5QQT5C7VTEDRAVCTZ7XWYZAFE52AYODCFJE74UQ, seg_pos={0 0}, upload=73230]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: context deadline exceeded; context deadline exceeded Sep 27 00:44:50.959914786 (83h13m36.422007269s) [seg_pos={0 0}, upload=73230]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 00:44:51.201781589 (83h13m36.663874072s) [upload=73230]: Done uploading segments... Sep 27 00:44:51.201868144 (83h13m36.663960617s) [upload=73230]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.919988077s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)