Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.614150237s FAIL (2025-10-24 09:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.614174506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.709333998s FAIL (2025-10-24 09:22:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.709386707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.607716283s FAIL (2025-10-24 09:21:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.60776717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.664580626s FAIL (2025-10-24 09:20:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.66482992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.659170637s FAIL (2025-10-24 09:19:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.659194322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.631795556s FAIL (2025-10-24 09:19:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.63201842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.645852454s FAIL (2025-10-24 09:18:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.645881598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.708267769s FAIL (2025-10-24 09:18:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.708301004s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.773555474s FAIL (2025-10-24 09:17:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.77357389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.718461245s FAIL (2025-10-24 09:16:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.718485562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.678979256s FAIL (2025-10-24 09:15:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.679232118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.562172867s FAIL (2025-10-24 09:14:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.562198806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.704427863s FAIL (2025-10-24 09:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.704707005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.00087776s FAIL (2025-10-24 09:13:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00090462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.062198674s FAIL (2025-10-24 07:14:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.062236395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.008996709s FAIL (2025-10-24 07:13:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.009047967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.954013405s FAIL (2025-10-24 07:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.95404266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.089284112s FAIL (2025-10-24 07:12:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.089312967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.122587779s FAIL (2025-10-24 07:11:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.122629318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.247160582s FAIL (2025-10-24 07:11:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.247198484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.994089259s FAIL (2025-10-24 07:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.994123505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.044713469s FAIL (2025-10-24 07:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.044738616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.052427022s FAIL (2025-10-24 07:09:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.052466397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.098647199s FAIL (2025-10-24 07:08:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.098679821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.058672661s FAIL (2025-10-24 07:07:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.05870351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.05890036s FAIL (2025-10-24 07:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.058933103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.017199235s FAIL (2025-10-24 07:06:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.017233531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.91615084s FAIL (2025-10-24 07:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.916182341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.106088146s FAIL (2025-10-24 07:05:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.106125947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.060497716s FAIL (2025-10-24 07:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.060534255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.085593224s FAIL (2025-10-24 07:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.085627428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.071139805s FAIL (2025-10-24 07:03:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.071174442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.981802897s FAIL (2025-10-24 07:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.981832393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.096127112s FAIL (2025-10-24 07:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.096163892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.127708559s FAIL (2025-10-24 07:01:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.127732113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.902320957s FAIL (2025-10-24 06:59:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.90234922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.029291384s FAIL (2025-10-24 06:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.029335577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.016208001s FAIL (2025-10-24 06:58:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.016256534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.025955743s FAIL (2025-10-24 06:57:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.025991872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.173133117s FAIL (2025-10-24 06:57:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.173160709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.110534448s FAIL (2025-10-24 06:56:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.110569555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.187956674s FAIL (2025-10-24 06:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.187982262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.099622512s FAIL (2025-10-24 06:55:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.099654943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.073434916s FAIL (2025-10-24 06:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.073486955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.91851514s FAIL (2025-10-24 06:53:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.91854685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.04476771s FAIL (2025-10-24 06:53:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.044802968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.090295382s FAIL (2025-10-24 06:52:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.090318786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.050213514s FAIL (2025-10-24 06:51:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.050247518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.05512499s FAIL (2025-10-24 06:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.055155617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.036033484s FAIL (2025-10-24 06:50:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 24 06:32:32.690437760 (737h1m18.152530243s) [upload=648254]: Starting upload Oct 24 06:32:32.690511100 (737h1m18.152603573s) [upload=648254]: Done starting upload Oct 24 06:32:32.690605459 (737h1m18.152697942s) [upload=648254]: Uploading segments... Oct 24 06:32:32.690685552 (737h1m18.152778055s) [upload=648254]: Got next segment. Inline: false Oct 24 06:32:32.690724075 (737h1m18.152816548s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 24 06:32:32.690730477 (737h1m18.15282296s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 24 06:32:32.945650786 (737h1m18.407743269s) [seg_pos={0 0}, upload=648254]: Begin upload segment... Oct 24 06:32:32.945722885 (737h1m18.407815368s) [seg_pos={0 0}, upload=648254]: Done begin upload segment. Oct 24 06:32:32.945758032 (737h1m18.407850515s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.945829748 (737h1m18.407922221s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946769928 (737h1m18.408862401s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.946967854 (737h1m18.409060338s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947199807 (737h1m18.40929228s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947381903 (737h1m18.409474386s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947544182 (737h1m18.409636655s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947760865 (737h1m18.409853338s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.947865454 (737h1m18.409957937s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948209088 (737h1m18.410301561s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948235429 (737h1m18.410327902s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948585315 (737h1m18.410677798s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.948710202 (737h1m18.410802675s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949104514 (737h1m18.411196997s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949273776 (737h1m18.411366249s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:32.949530154 (737h1m18.411622637s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Uploading piece... Oct 24 06:32:34.629734333 (737h1m20.091826816s) [upload=648254]: Next returned nil segment Oct 24 06:32:34.629761184 (737h1m20.091853657s) [upload=648254]: Waiting for error group managing segments... Oct 24 06:32:35.028511386 (737h1m20.490603869s) [noise=true , address=145.79.227.23:4545 , piece_id=XWBC5GWXD6ETEJVVQTEAIAVKESF7UH2EF6ZHOXX6ARDTB5ZS5DIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.031517369 (737h1m20.493609842s) [noise=true , address=145.79.227.4:4185 , piece_id=7C6JSKJSXIPYZCEUEHZ6XT6EWHAENBTHGB4TZMPK3ZJVKKMNUJSA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.042908240 (737h1m20.505000723s) [noise=true , address=145.79.227.19:4079 , piece_id=UMFZV5O3AX7QPKO35T7CHSDV23XVQWSP4PAZAQT7CBSPSYEHEYIA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.046380371 (737h1m20.508472854s) [noise=true , address=45.91.116.10:4082 , piece_id=RQXMXKB4IQWI5XCECBKQRLPAORY7LH3HTFEG6A5HDACXVPFZHCEA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.047793783 (737h1m20.509886266s) [noise=true , address=145.79.227.14:4840 , piece_id=WOGI776B76Z7DF3VO6AHUQSWEU6SX32JPTKFXE4EIIARBQXTI47A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.050048376 (737h1m20.512140859s) [noise=true , address=145.79.227.20:4438 , piece_id=VTNWOEPONFYNQQR4DUO65Q3UZRUL35LL4JAN2AYIPHVRHVKYIFVQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.051898769 (737h1m20.513991252s) [noise=true , address=77.90.31.15:4699 , piece_id=PKJW4ZR3XIBOFN3UKXJYYYEWVKMP2QWMARL6FXZ4KGQJY5OEM7XA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061017363 (737h1m20.523109846s) [noise=true , address=145.79.227.10:4402 , piece_id=EJTUBKREJLMZTTCEIKALYZZ5CHFLNSPG5CETMGQOQOE73KDCJSLQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.061598229 (737h1m20.523690712s) [noise=true , address=145.79.227.21:4475 , piece_id=CW7QXTL6MUPU33AEROA5TNZYVOVNUE76HURBCKIEBQMREEMBP7CA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.066181998 (737h1m20.528274481s) [noise=true , address=145.79.227.18:4007 , piece_id=JTYSIW5YRLNV4WHVVOISCOTA2R7KNDTQBEJIO2CCV7SN2IJAO74A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.078697851 (737h1m20.540790334s) [noise=true , address=145.79.227.22:4511 , piece_id=KATIOMSPMAXHCNI4YC3QMXK5QSO4FHSFVQDWRL5MX7WIDCONIPUA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.083031562 (737h1m20.545124045s) [noise=true , address=145.79.227.21:4474 , piece_id=Z5SDJNDBXHFZJRMCWDYR3C7Y26TGFHDWPOFUX2NWY3J2Y7VYW5QQ, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086970604 (737h1m20.549063087s) [noise=true , address=145.79.227.23:4547 , piece_id=3UCZ5HO2IRV6NEANWJUTDHSVNOHVJOQ5GXGMTFJCKULWL6R246FA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: <nil> Oct 24 06:32:35.086999719 (737h1m20.549092192s) [seg_pos={0 0}, upload=648254]: Segment reached optimal threshold of 13 pieces. Oct 24 06:32:35.087136229 (737h1m20.549228722s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087147170 (737h1m20.549239653s) [noise=true , address=145.79.227.21:4473 , piece_id=5H32ELLDFREGD4PEATCM7DCIOGMTOSMGZY3OMFPUWPF4RRRLEBHA, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087304840 (737h1m20.549397323s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47650->145.79.227.7:4295: use of closed network connection Oct 24 06:32:35.087314919 (737h1m20.549407403s) [noise=true , address=145.79.227.7:4295 , piece_id=JC6625L7SGZHMSQJO3XTFWI334NQUIYIWAZYAZGYTHJJLJ7QL24A, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087403988 (737h1m20.549496472s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Oct 24 06:32:35.087411162 (737h1m20.549503645s) [noise=true , address=45.91.116.9:4263 , piece_id=2UFWANKAIP36LBSLZGXYXFY6WAGNYQMAOVEXXN4BMZ3YTCYNIA5Q, seg_pos={0 0}, upload=648254]: Long tail cancellation due to optimal threshold Oct 24 06:32:35.087428425 (737h1m20.549520898s) [seg_pos={0 0}, upload=648254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 24 06:32:35.087511203 (737h1m20.549603676s) [upload=648254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 24 06:32:35.087518517 (737h1m20.549611s) [upload=648254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 24 06:33:13.032738764 (737h1m58.494831277s) [upload=648254]: Done uploading segments... Oct 24 06:33:13.032785033 (737h1m58.494877516s) [upload=648254]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.036069373s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)