Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXXXX......X.X.X.X.X..............XX..X..X...........XXX.X.] duration: 6.144699186s trace: https://jaeger.dev.storj.io/trace/7eb8c3dc1d9f462 FAIL (2025-12-26 20:14:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ) context deadline exceeded ======LOGS====== c 26 20:13:30.255904555 (508h44m30.1269876s) [upload=446422]: Starting upload Dec 26 20:13:30.256022840 (508h44m30.127105885s) [upload=446422]: Done starting upload Dec 26 20:13:30.256105388 (508h44m30.127188423s) [upload=446422]: Uploading segments... Dec 26 20:13:30.256283427 (508h44m30.127366472s) [upload=446422]: Got next segment. Inline: false Dec 26 20:13:30.256353240 (508h44m30.127436275s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:13:30.256360033 (508h44m30.127443068s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:13:30.504911562 (508h44m30.375994618s) [seg_pos={0 0}, upload=446422]: Begin upload segment... Dec 26 20:13:30.504968291 (508h44m30.376051326s) [seg_pos={0 0}, upload=446422]: Done begin upload segment. Dec 26 20:13:30.505009940 (508h44m30.376093005s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505068542 (508h44m30.376151577s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505857063 (508h44m30.376940118s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505923760 (508h44m30.377006805s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506292352 (508h44m30.377375387s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506745395 (508h44m30.37782844s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507126350 (508h44m30.378209816s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507493379 (508h44m30.378576414s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507837003 (508h44m30.378920058s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508355170 (508h44m30.379438205s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508734031 (508h44m30.379817076s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509153139 (508h44m30.380236174s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509564182 (508h44m30.380647217s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509940668 (508h44m30.381023703s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.510990307 (508h44m30.382073342s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.511425135 (508h44m30.38250817s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:32.214275616 (508h44m32.085358651s) [upload=446422]: Next returned nil segment Dec 26 20:13:32.214291817 (508h44m32.085374852s) [upload=446422]: Waiting for error group managing segments... Dec 26 20:13:32.508523589 (508h44m32.379606635s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.509628013 (508h44m32.380711048s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.512947394 (508h44m32.384030429s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.513373014 (508h44m32.384456049s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.518879399 (508h44m32.389962434s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.524877440 (508h44m32.395960485s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.528606681 (508h44m32.399689716s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.532434430 (508h44m32.403517475s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.540544673 (508h44m32.411627708s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.000249101 (508h44m32.871332146s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.775658741 (508h44m33.646741786s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.803403391 (508h44m33.674486427s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138384406 (508h44m34.009467441s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138424493 (508h44m34.009507538s) [seg_pos={0 0}, upload=446422]: Segment reached optimal threshold of 13 pieces. Dec 26 20:13:34.138777035 (508h44m34.00986008s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138787494 (508h44m34.009870529s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.138891713 (508h44m34.009974748s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138902623 (508h44m34.009985668s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139071996 (508h44m34.010155041s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection Dec 26 20:13:34.139080772 (508h44m34.010163807s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139103416 (508h44m34.010186451s) [seg_pos={0 0}, upload=446422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:13:34.139181664 (508h44m34.01026471s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:13:34.139189811 (508h44m34.010272886s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:13:34.440108426 (508h44m34.311191471s) [upload=446422]: Done uploading segments... Dec 26 20:13:34.440126200 (508h44m34.311209235s) [upload=446422]: Upload finished. err: <nil> Duration: 6.144718041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XXXXX......X.X.X.X.X..............XX..X..X...........XXX.X] duration: 30.003124896s trace: https://jaeger.dev.storj.io/trace/7eb8c3dc1d9f462 FAIL (2025-12-26 20:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ) context deadline exceeded ======LOGS====== c 26 20:13:30.255904555 (508h44m30.1269876s) [upload=446422]: Starting upload Dec 26 20:13:30.256022840 (508h44m30.127105885s) [upload=446422]: Done starting upload Dec 26 20:13:30.256105388 (508h44m30.127188423s) [upload=446422]: Uploading segments... Dec 26 20:13:30.256283427 (508h44m30.127366472s) [upload=446422]: Got next segment. Inline: false Dec 26 20:13:30.256353240 (508h44m30.127436275s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:13:30.256360033 (508h44m30.127443068s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:13:30.504911562 (508h44m30.375994618s) [seg_pos={0 0}, upload=446422]: Begin upload segment... Dec 26 20:13:30.504968291 (508h44m30.376051326s) [seg_pos={0 0}, upload=446422]: Done begin upload segment. Dec 26 20:13:30.505009940 (508h44m30.376093005s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505068542 (508h44m30.376151577s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505857063 (508h44m30.376940118s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505923760 (508h44m30.377006805s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506292352 (508h44m30.377375387s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506745395 (508h44m30.37782844s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507126350 (508h44m30.378209816s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507493379 (508h44m30.378576414s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507837003 (508h44m30.378920058s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508355170 (508h44m30.379438205s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508734031 (508h44m30.379817076s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509153139 (508h44m30.380236174s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509564182 (508h44m30.380647217s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509940668 (508h44m30.381023703s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.510990307 (508h44m30.382073342s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.511425135 (508h44m30.38250817s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:32.214275616 (508h44m32.085358651s) [upload=446422]: Next returned nil segment Dec 26 20:13:32.214291817 (508h44m32.085374852s) [upload=446422]: Waiting for error group managing segments... Dec 26 20:13:32.508523589 (508h44m32.379606635s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.509628013 (508h44m32.380711048s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.512947394 (508h44m32.384030429s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.513373014 (508h44m32.384456049s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.518879399 (508h44m32.389962434s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.524877440 (508h44m32.395960485s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.528606681 (508h44m32.399689716s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.532434430 (508h44m32.403517475s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.540544673 (508h44m32.411627708s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.000249101 (508h44m32.871332146s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.775658741 (508h44m33.646741786s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.803403391 (508h44m33.674486427s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138384406 (508h44m34.009467441s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138424493 (508h44m34.009507538s) [seg_pos={0 0}, upload=446422]: Segment reached optimal threshold of 13 pieces. Dec 26 20:13:34.138777035 (508h44m34.00986008s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138787494 (508h44m34.009870529s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.138891713 (508h44m34.009974748s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138902623 (508h44m34.009985668s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139071996 (508h44m34.010155041s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection Dec 26 20:13:34.139080772 (508h44m34.010163807s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139103416 (508h44m34.010186451s) [seg_pos={0 0}, upload=446422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:13:34.139181664 (508h44m34.01026471s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:13:34.139189811 (508h44m34.010272886s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:13:34.440108426 (508h44m34.311191471s) [upload=446422]: Done uploading segments... Dec 26 20:13:34.440126200 (508h44m34.311209235s) [upload=446422]: Upload finished. err: <nil> Duration: 30.003156819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 15.422832588s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:13:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 15.422852145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 10.350005706s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 10.350035484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.XXXXX......X.X.X.X.X..............XX..X..X...........XXX.] duration: 7.3220736s trace: https://jaeger.dev.storj.io/trace/60a742e03d62b261 FAIL (2025-12-26 20:12:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA) context deadline exceeded ======LOGS====== Dec 26 20:11:18.441997655 (508h42m18.31308069s) [upload=446389]: Starting upload Dec 26 20:11:18.442091924 (508h42m18.313174969s) [upload=446389]: Done starting upload Dec 26 20:11:18.442158060 (508h42m18.313241095s) [upload=446389]: Uploading segments... Dec 26 20:11:18.442328134 (508h42m18.313411179s) [upload=446389]: Got next segment. Inline: false Dec 26 20:11:18.442376515 (508h42m18.31345956s) [upload=446389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:11:18.442381686 (508h42m18.313464721s) [upload=446389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:11:18.689641056 (508h42m18.560724101s) [seg_pos={0 0}, upload=446389]: Begin upload segment... Dec 26 20:11:18.689752728 (508h42m18.560835763s) [seg_pos={0 0}, upload=446389]: Done begin upload segment. Dec 26 20:11:18.690031258 (508h42m18.561114293s) [noise=true , address=145.79.227.52:4447 , piece_id=IGMPKDXQ2TZZHWQD7ESFYSI6M2TKGLYVHHXTJPHYRX6SUHRHYK4A, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.690259063 (508h42m18.561342098s) [noise=true , address=145.79.227.45:4230 , piece_id=DLUPTUGHMUF5TAPRWX63LSQ4FSKHH7D2KZIB2WLNRYHFH3NPBUQA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691007597 (508h42m18.562090633s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691139198 (508h42m18.562222233s) [noise=true , address=145.79.227.11:4731 , piece_id=HME46CLCATCJUGBAEDW3WMEOXDZLWX6GPVAE5M43U4LMUQP63EKA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691428960 (508h42m18.562511995s) [noise=true , address=145.79.227.39:4014 , piece_id=EAZNGILPW4WKWNA6X5BZICKEC6HHB664TTYIZQGEVBXMZKXQW77A, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691438378 (508h42m18.562521423s) [noise=true , address=145.79.227.25:4618 , piece_id=D2DEAAOXBUM36I2DTWWYMVABVPZ4FJR5R72EQYIML5RWFGA7YKAQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691726507 (508h42m18.562809542s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691811688 (508h42m18.562894734s) [noise=true , address=145.79.227.29:4942 , piece_id=NJDXXPGPDQOW6P2GSOTGFYDTWPHKTKLA5A4LNTCCGN4HJEFIZ7LQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691976933 (508h42m18.563059968s) [noise=true , address=77.90.31.16:4736 , piece_id=2V5KNISGQMHALS2WUPFIRRV62YYYAIRALNR5PTLPWYTAO6HEECCQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692248741 (508h42m18.563331806s) [noise=true , address=45.91.116.13:4153 , piece_id=ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692273749 (508h42m18.563356794s) [noise=true , address=45.91.116.8:4046 , piece_id=YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692572157 (508h42m18.563655192s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692678860 (508h42m18.563761895s) [noise=true , address=45.91.116.12:4225 , piece_id=ZWVFH6PXPBHJYZYFKKFG5JIRNBROZGQWIEOPVWIWQEE3XOZXKA7Q, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692834877 (508h42m18.563917912s) [noise=true , address=45.91.116.10:4084 , piece_id=YMZI76SXDHFJX62MXTF4JMJX2S25UJRUXHSRKY6D7HKHT3NV6CDQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.693106033 (508h42m18.564189068s) [noise=true , address=45.91.116.11:4118 , piece_id=RMI6PFALCUAS7SYNIEH7PUBUBRFH6NCZA2LM22PORNN2KHJW3NZQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.693416204 (508h42m18.564499239s) [noise=true , address=145.79.227.51:4409 , piece_id=MNSHAMBQ64N5KTRU4U743XD7HYI42BZB6KLA4G4ZIAPE7A2KYAKQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:20.314601612 (508h42m20.185684657s) [upload=446389]: Next returned nil segment Dec 26 20:11:20.314625497 (508h42m20.185708532s) [upload=446389]: Waiting for error group managing segments... Dec 26 20:11:20.697656308 (508h42m20.568739353s) [noise=true , address=145.79.227.11:4731 , piece_id=HME46CLCATCJUGBAEDW3WMEOXDZLWX6GPVAE5M43U4LMUQP63EKA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.702368099 (508h42m20.573451144s) [noise=true , address=145.79.227.52:4447 , piece_id=IGMPKDXQ2TZZHWQD7ESFYSI6M2TKGLYVHHXTJPHYRX6SUHRHYK4A, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.706556735 (508h42m20.5776398s) [noise=true , address=145.79.227.45:4230 , piece_id=DLUPTUGHMUF5TAPRWX63LSQ4FSKHH7D2KZIB2WLNRYHFH3NPBUQA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.716725628 (508h42m20.587808663s) [noise=true , address=145.79.227.51:4409 , piece_id=MNSHAMBQ64N5KTRU4U743XD7HYI42BZB6KLA4G4ZIAPE7A2KYAKQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.717120820 (508h42m20.588203855s) [noise=true , address=145.79.227.39:4014 , piece_id=EAZNGILPW4WKWNA6X5BZICKEC6HHB664TTYIZQGEVBXMZKXQW77A, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.722097216 (508h42m20.593180281s) [noise=true , address=145.79.227.25:4618 , piece_id=D2DEAAOXBUM36I2DTWWYMVABVPZ4FJR5R72EQYIML5RWFGA7YKAQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.723703845 (508h42m20.5947869s) [noise=true , address=145.79.227.29:4942 , piece_id=NJDXXPGPDQOW6P2GSOTGFYDTWPHKTKLA5A4LNTCCGN4HJEFIZ7LQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.903294054 (508h42m20.774377089s) [noise=true , address=77.90.31.16:4736 , piece_id=2V5KNISGQMHALS2WUPFIRRV62YYYAIRALNR5PTLPWYTAO6HEECCQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.926202725 (508h42m20.79728577s) [noise=true , address=45.91.116.13:4153 , piece_id=ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.942030142 (508h42m20.813113188s) [noise=true , address=45.91.116.10:4084 , piece_id=YMZI76SXDHFJX62MXTF4JMJX2S25UJRUXHSRKY6D7HKHT3NV6CDQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.975716748 (508h42m20.846799783s) [noise=true , address=45.91.116.12:4225 , piece_id=ZWVFH6PXPBHJYZYFKKFG5JIRNBROZGQWIEOPVWIWQEE3XOZXKA7Q, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.997141584 (508h42m20.868224629s) [noise=true , address=45.91.116.11:4118 , piece_id=RMI6PFALCUAS7SYNIEH7PUBUBRFH6NCZA2LM22PORNN2KHJW3NZQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:21.052475598 (508h42m20.923558633s) [noise=true , address=45.91.116.8:4046 , piece_id=YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:21.052494945 (508h42m20.92357799s) [seg_pos={0 0}, upload=446389]: Segment reached optimal threshold of 13 pieces. Dec 26 20:11:21.052769719 (508h42m20.923852764s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 20:11:21.052779978 (508h42m20.923863033s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.052954039 (508h42m20.924037074s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 20:11:21.052962455 (508h42m20.92404549s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.052973356 (508h42m20.924056392s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection Dec 26 20:11:21.052980911 (508h42m20.924063956s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.053007612 (508h42m20.924090647s) [seg_pos={0 0}, upload=446389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:11:21.053097081 (508h42m20.924180127s) [upload=446389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:11:21.053105187 (508h42m20.924188222s) [upload=446389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:11:21.356469750 (508h42m21.227552815s) [upload=446389]: Done uploading segments... Dec 26 20:11:21.356496531 (508h42m21.227579576s) [upload=446389]: Upload finished. err: <nil> Duration: 7.322097504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 32.379450705s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:12:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 32.379466726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.XXXXX......X.X.X.X.X..............XX..X..X...........XXX] duration: 30.003492065s trace: https://jaeger.dev.storj.io/trace/60a742e03d62b261 FAIL (2025-12-26 20:11:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA) context deadline exceeded ======LOGS====== Dec 26 20:11:18.441997655 (508h42m18.31308069s) [upload=446389]: Starting upload Dec 26 20:11:18.442091924 (508h42m18.313174969s) [upload=446389]: Done starting upload Dec 26 20:11:18.442158060 (508h42m18.313241095s) [upload=446389]: Uploading segments... Dec 26 20:11:18.442328134 (508h42m18.313411179s) [upload=446389]: Got next segment. Inline: false Dec 26 20:11:18.442376515 (508h42m18.31345956s) [upload=446389]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:11:18.442381686 (508h42m18.313464721s) [upload=446389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:11:18.689641056 (508h42m18.560724101s) [seg_pos={0 0}, upload=446389]: Begin upload segment... Dec 26 20:11:18.689752728 (508h42m18.560835763s) [seg_pos={0 0}, upload=446389]: Done begin upload segment. Dec 26 20:11:18.690031258 (508h42m18.561114293s) [noise=true , address=145.79.227.52:4447 , piece_id=IGMPKDXQ2TZZHWQD7ESFYSI6M2TKGLYVHHXTJPHYRX6SUHRHYK4A, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.690259063 (508h42m18.561342098s) [noise=true , address=145.79.227.45:4230 , piece_id=DLUPTUGHMUF5TAPRWX63LSQ4FSKHH7D2KZIB2WLNRYHFH3NPBUQA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691007597 (508h42m18.562090633s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691139198 (508h42m18.562222233s) [noise=true , address=145.79.227.11:4731 , piece_id=HME46CLCATCJUGBAEDW3WMEOXDZLWX6GPVAE5M43U4LMUQP63EKA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691428960 (508h42m18.562511995s) [noise=true , address=145.79.227.39:4014 , piece_id=EAZNGILPW4WKWNA6X5BZICKEC6HHB664TTYIZQGEVBXMZKXQW77A, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691438378 (508h42m18.562521423s) [noise=true , address=145.79.227.25:4618 , piece_id=D2DEAAOXBUM36I2DTWWYMVABVPZ4FJR5R72EQYIML5RWFGA7YKAQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691726507 (508h42m18.562809542s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691811688 (508h42m18.562894734s) [noise=true , address=145.79.227.29:4942 , piece_id=NJDXXPGPDQOW6P2GSOTGFYDTWPHKTKLA5A4LNTCCGN4HJEFIZ7LQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.691976933 (508h42m18.563059968s) [noise=true , address=77.90.31.16:4736 , piece_id=2V5KNISGQMHALS2WUPFIRRV62YYYAIRALNR5PTLPWYTAO6HEECCQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692248741 (508h42m18.563331806s) [noise=true , address=45.91.116.13:4153 , piece_id=ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692273749 (508h42m18.563356794s) [noise=true , address=45.91.116.8:4046 , piece_id=YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692572157 (508h42m18.563655192s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692678860 (508h42m18.563761895s) [noise=true , address=45.91.116.12:4225 , piece_id=ZWVFH6PXPBHJYZYFKKFG5JIRNBROZGQWIEOPVWIWQEE3XOZXKA7Q, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.692834877 (508h42m18.563917912s) [noise=true , address=45.91.116.10:4084 , piece_id=YMZI76SXDHFJX62MXTF4JMJX2S25UJRUXHSRKY6D7HKHT3NV6CDQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.693106033 (508h42m18.564189068s) [noise=true , address=45.91.116.11:4118 , piece_id=RMI6PFALCUAS7SYNIEH7PUBUBRFH6NCZA2LM22PORNN2KHJW3NZQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:18.693416204 (508h42m18.564499239s) [noise=true , address=145.79.227.51:4409 , piece_id=MNSHAMBQ64N5KTRU4U743XD7HYI42BZB6KLA4G4ZIAPE7A2KYAKQ, seg_pos={0 0}, upload=446389]: Uploading piece... Dec 26 20:11:20.314601612 (508h42m20.185684657s) [upload=446389]: Next returned nil segment Dec 26 20:11:20.314625497 (508h42m20.185708532s) [upload=446389]: Waiting for error group managing segments... Dec 26 20:11:20.697656308 (508h42m20.568739353s) [noise=true , address=145.79.227.11:4731 , piece_id=HME46CLCATCJUGBAEDW3WMEOXDZLWX6GPVAE5M43U4LMUQP63EKA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.702368099 (508h42m20.573451144s) [noise=true , address=145.79.227.52:4447 , piece_id=IGMPKDXQ2TZZHWQD7ESFYSI6M2TKGLYVHHXTJPHYRX6SUHRHYK4A, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.706556735 (508h42m20.5776398s) [noise=true , address=145.79.227.45:4230 , piece_id=DLUPTUGHMUF5TAPRWX63LSQ4FSKHH7D2KZIB2WLNRYHFH3NPBUQA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.716725628 (508h42m20.587808663s) [noise=true , address=145.79.227.51:4409 , piece_id=MNSHAMBQ64N5KTRU4U743XD7HYI42BZB6KLA4G4ZIAPE7A2KYAKQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.717120820 (508h42m20.588203855s) [noise=true , address=145.79.227.39:4014 , piece_id=EAZNGILPW4WKWNA6X5BZICKEC6HHB664TTYIZQGEVBXMZKXQW77A, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.722097216 (508h42m20.593180281s) [noise=true , address=145.79.227.25:4618 , piece_id=D2DEAAOXBUM36I2DTWWYMVABVPZ4FJR5R72EQYIML5RWFGA7YKAQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.723703845 (508h42m20.5947869s) [noise=true , address=145.79.227.29:4942 , piece_id=NJDXXPGPDQOW6P2GSOTGFYDTWPHKTKLA5A4LNTCCGN4HJEFIZ7LQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.903294054 (508h42m20.774377089s) [noise=true , address=77.90.31.16:4736 , piece_id=2V5KNISGQMHALS2WUPFIRRV62YYYAIRALNR5PTLPWYTAO6HEECCQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.926202725 (508h42m20.79728577s) [noise=true , address=45.91.116.13:4153 , piece_id=ORO77TCJZ6W73C4KHNBNFMPHN3FTRM3R7IUILQ2VF6FXOXTHDIEA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.942030142 (508h42m20.813113188s) [noise=true , address=45.91.116.10:4084 , piece_id=YMZI76SXDHFJX62MXTF4JMJX2S25UJRUXHSRKY6D7HKHT3NV6CDQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.975716748 (508h42m20.846799783s) [noise=true , address=45.91.116.12:4225 , piece_id=ZWVFH6PXPBHJYZYFKKFG5JIRNBROZGQWIEOPVWIWQEE3XOZXKA7Q, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:20.997141584 (508h42m20.868224629s) [noise=true , address=45.91.116.11:4118 , piece_id=RMI6PFALCUAS7SYNIEH7PUBUBRFH6NCZA2LM22PORNN2KHJW3NZQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:21.052475598 (508h42m20.923558633s) [noise=true , address=45.91.116.8:4046 , piece_id=YXYSC6O37ER7OXPM7MV6JG4VTOEKIEVLD4XLOCHMHIT4XJPF3YZA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: <nil> Dec 26 20:11:21.052494945 (508h42m20.92357799s) [seg_pos={0 0}, upload=446389]: Segment reached optimal threshold of 13 pieces. Dec 26 20:11:21.052769719 (508h42m20.923852764s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 20:11:21.052779978 (508h42m20.923863033s) [noise=true , address=145.79.227.34:4906 , piece_id=TCNX4ECMWSVP7SNK66X4VKRNNOGKTFYE7ZSQBHUWZQZQ5JVYJFYA, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.052954039 (508h42m20.924037074s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 20:11:21.052962455 (508h42m20.92404549s) [noise=true , address=145.79.227.44:4194 , piece_id=5DURW3S4P3T277YPXVJRDM77Z64D5EE3XKYKGZQGOI26HCXKF7UA, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.052973356 (508h42m20.924056392s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48812->77.90.31.11:4554: use of closed network connection Dec 26 20:11:21.052980911 (508h42m20.924063956s) [noise=true , address=77.90.31.11:4554 , piece_id=RNYWM34J2IWNINDDARA543Z3DICDSITK6VSVLVTBTJJZB2ERVYOQ, seg_pos={0 0}, upload=446389]: Long tail cancellation due to optimal threshold Dec 26 20:11:21.053007612 (508h42m20.924090647s) [seg_pos={0 0}, upload=446389]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:11:21.053097081 (508h42m20.924180127s) [upload=446389]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:11:21.053105187 (508h42m20.924188222s) [upload=446389]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:11:21.356469750 (508h42m21.227552815s) [upload=446389]: Done uploading segments... Dec 26 20:11:21.356496531 (508h42m21.227579576s) [upload=446389]: Upload finished. err: <nil> Duration: 30.003518957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 22.085158334s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 22.085183121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 29.858112695s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 29.858155856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.XXXXX......X.X.X.X.X..............XX..X..X...........XX] duration: 30.003061558s trace: https://jaeger.dev.storj.io/trace/4821c6afc56390d5 FAIL (2025-12-26 20:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 57336925s) [upload=446369]: Uploading segments... Dec 26 20:10:04.586405598 (508h41m4.457488643s) [upload=446369]: Got next segment. Inline: false Dec 26 20:10:04.586680832 (508h41m4.457763878s) [upload=446369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:10:04.586691654 (508h41m4.457774699s) [upload=446369]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:10:04.835479844 (508h41m4.706562889s) [seg_pos={0 0}, upload=446369]: Begin upload segment... Dec 26 20:10:04.835564715 (508h41m4.70664776s) [seg_pos={0 0}, upload=446369]: Done begin upload segment. Dec 26 20:10:04.835636311 (508h41m4.706719356s) [noise=true , address=145.79.227.36:4518 , piece_id=6JMN57KNWHXMZDRTBXWNC7OJVBIHQ45TD5IF5G5M4G65QQ5AEOBQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.835675986 (508h41m4.706759031s) [noise=true , address=45.91.116.12:4226 , piece_id=C37AXGTTQYFRM7UW35TBIZERZKTPCO7CVHEMHKD6XA2SRK2FZTHA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.836353948 (508h41m4.707436993s) [noise=true , address=45.91.116.10:4081 , piece_id=V33SVD3EBWNTYZUKSISIGTBOH7V2ZUWV3AYZLFBS57E2757JT3DQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.836905238 (508h41m4.707988283s) [noise=true , address=145.79.227.37:4566 , piece_id=453BHY3EUO4ZKTEIBX3355JJMIXY2B526LBEGFJJTSHV53FDAI4Q, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.837383429 (508h41m4.708466474s) [noise=true , address=145.79.227.5:4221 , piece_id=RUYM3EDS5HPSFDVPZERUSJBGQNYJVVMV362M5O66MLC3E7S6YQAA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.837788199 (508h41m4.708871244s) [noise=true , address=145.79.227.11:4732 , piece_id=66A4VOGQ7VYNB3SMSLROPIKBFTQCZTE6AF2GSNMO2LL3TSHODOYA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.838366300 (508h41m4.709449335s) [noise=true , address=145.79.227.17:4972 , piece_id=FRUNFPATXZMRGLHMBZHQNTJIHLPJTJ5LANDPZ3K4GG6U4GYH7WNQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.838967204 (508h41m4.710050249s) [noise=true , address=145.79.227.25:4618 , piece_id=7PXWZLX2BM66MDYB2756SQERSHWMGHAJKPCVTWEQ6IVRP22Z3CSQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.840337282 (508h41m4.711420328s) [noise=true , address=145.79.227.23:4547 , piece_id=V4MDKBVMCUIUAOGBH2N5BIN3IFGUCEODZKIAFGB5FUPCFIXCRGMA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.840542203 (508h41m4.711625248s) [noise=true , address=45.91.116.9:4261 , piece_id=3BVYNPWTSLC6QMRQWNIL3KBBTXAOLJ3CMDZOAYOOLPLCWTORABDQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.841940946 (508h41m4.713023991s) [noise=true , address=145.79.227.38:4977 , piece_id=HDFVJKN2NG63EDRAKRT6SHQSANWMMNTQFJAD2GEOVY4RGLCMFEGQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.842951942 (508h41m4.714034997s) [noise=true , address=145.79.227.49:4338 , piece_id=53W5D7JSWHJKXHASORQVXXKTNGBUQRJZEABRVBKTJKL23XQ6KA3Q, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.843397770 (508h41m4.714480815s) [noise=true , address=45.91.116.5:4300 , piece_id=2L46BBXVQ44JKF6CFVOQUYCVJBTQ5PGEIZAU5VSENI2EPVQLI4XA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.844756988 (508h41m4.715840033s) [noise=true , address=145.79.227.52:4447 , piece_id=MGZBEXENAYWCCQSEQRP2DUVMO5IP3KAI7VNSQBWKTU5KQIZ5LTIQ, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.846212380 (508h41m4.717295415s) [noise=true , address=145.79.227.50:4374 , piece_id=YQESMJCP4SX3YP5OGGWYL5XPPU7R6573IPTQDWCG5V37CNOLIB5A, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:04.846716821 (508h41m4.717799856s) [noise=true , address=145.79.227.42:4122 , piece_id=B2S6FELFQB4WX77FULQFNIOZEQQKW3HUW6WJOY5XIFUJME7M3GBA, seg_pos={0 0}, upload=446369]: Uploading piece... Dec 26 20:10:06.465171990 (508h41m6.336255045s) [upload=446369]: Next returned nil segment Dec 26 20:10:06.465189544 (508h41m6.336272599s) [upload=446369]: Waiting for error group managing segments... Dec 26 20:10:06.847884574 (508h41m6.718967609s) [noise=true , address=145.79.227.52:4447 , piece_id=MGZBEXENAYWCCQSEQRP2DUVMO5IP3KAI7VNSQBWKTU5KQIZ5LTIQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.848675230 (508h41m6.719758275s) [noise=true , address=145.79.227.23:4547 , piece_id=V4MDKBVMCUIUAOGBH2N5BIN3IFGUCEODZKIAFGB5FUPCFIXCRGMA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.849944997 (508h41m6.721028042s) [noise=true , address=145.79.227.50:4374 , piece_id=YQESMJCP4SX3YP5OGGWYL5XPPU7R6573IPTQDWCG5V37CNOLIB5A, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.850431533 (508h41m6.721514579s) [noise=true , address=145.79.227.37:4566 , piece_id=453BHY3EUO4ZKTEIBX3355JJMIXY2B526LBEGFJJTSHV53FDAI4Q, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.851295008 (508h41m6.722378043s) [noise=true , address=145.79.227.49:4338 , piece_id=53W5D7JSWHJKXHASORQVXXKTNGBUQRJZEABRVBKTJKL23XQ6KA3Q, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.859103116 (508h41m6.730186171s) [noise=true , address=145.79.227.38:4977 , piece_id=HDFVJKN2NG63EDRAKRT6SHQSANWMMNTQFJAD2GEOVY4RGLCMFEGQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.864720963 (508h41m6.735803998s) [noise=true , address=145.79.227.36:4518 , piece_id=6JMN57KNWHXMZDRTBXWNC7OJVBIHQ45TD5IF5G5M4G65QQ5AEOBQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.874544529 (508h41m6.745627574s) [noise=true , address=145.79.227.42:4122 , piece_id=B2S6FELFQB4WX77FULQFNIOZEQQKW3HUW6WJOY5XIFUJME7M3GBA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.877468256 (508h41m6.748551301s) [noise=true , address=145.79.227.25:4618 , piece_id=7PXWZLX2BM66MDYB2756SQERSHWMGHAJKPCVTWEQ6IVRP22Z3CSQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:06.894039079 (508h41m6.765122124s) [noise=true , address=145.79.227.11:4732 , piece_id=66A4VOGQ7VYNB3SMSLROPIKBFTQCZTE6AF2GSNMO2LL3TSHODOYA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:07.123576613 (508h41m6.994659658s) [noise=true , address=45.91.116.5:4300 , piece_id=2L46BBXVQ44JKF6CFVOQUYCVJBTQ5PGEIZAU5VSENI2EPVQLI4XA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:07.168426575 (508h41m7.03950962s) [noise=true , address=45.91.116.9:4261 , piece_id=3BVYNPWTSLC6QMRQWNIL3KBBTXAOLJ3CMDZOAYOOLPLCWTORABDQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:07.184111302 (508h41m7.055194337s) [noise=true , address=45.91.116.12:4226 , piece_id=C37AXGTTQYFRM7UW35TBIZERZKTPCO7CVHEMHKD6XA2SRK2FZTHA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: <nil> Dec 26 20:10:07.184139625 (508h41m7.05522266s) [seg_pos={0 0}, upload=446369]: Segment reached optimal threshold of 13 pieces. Dec 26 20:10:07.184392106 (508h41m7.055475152s) [noise=true , address=45.91.116.10:4081 , piece_id=V33SVD3EBWNTYZUKSISIGTBOH7V2ZUWV3AYZLFBS57E2757JT3DQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 20:10:07.184401855 (508h41m7.05548489s) [noise=true , address=45.91.116.10:4081 , piece_id=V33SVD3EBWNTYZUKSISIGTBOH7V2ZUWV3AYZLFBS57E2757JT3DQ, seg_pos={0 0}, upload=446369]: Long tail cancellation due to optimal threshold Dec 26 20:10:07.184640099 (508h41m7.055723134s) [noise=true , address=145.79.227.5:4221 , piece_id=RUYM3EDS5HPSFDVPZERUSJBGQNYJVVMV362M5O66MLC3E7S6YQAA, seg_pos={0 0}, upload=446369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:46584->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:46584->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46584->145.79.227.5:4221: use of closed network connection Dec 26 20:10:07.184647202 (508h41m7.055730247s) [noise=true , address=145.79.227.5:4221 , piece_id=RUYM3EDS5HPSFDVPZERUSJBGQNYJVVMV362M5O66MLC3E7S6YQAA, seg_pos={0 0}, upload=446369]: Long tail cancellation due to optimal threshold Dec 26 20:10:07.184801966 (508h41m7.055885011s) [noise=true , address=145.79.227.17:4972 , piece_id=FRUNFPATXZMRGLHMBZHQNTJIHLPJTJ5LANDPZ3K4GG6U4GYH7WNQ, seg_pos={0 0}, upload=446369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:35732->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:35732->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35732->145.79.227.17:4972: use of closed network connection Dec 26 20:10:07.184822596 (508h41m7.055905631s) [noise=true , address=145.79.227.17:4972 , piece_id=FRUNFPATXZMRGLHMBZHQNTJIHLPJTJ5LANDPZ3K4GG6U4GYH7WNQ, seg_pos={0 0}, upload=446369]: Long tail cancellation due to optimal threshold Dec 26 20:10:07.184874956 (508h41m7.055957991s) [seg_pos={0 0}, upload=446369]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:10:07.185025853 (508h41m7.056108898s) [upload=446369]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:10:07.185033487 (508h41m7.056116522s) [upload=446369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:10:07.487520048 (508h41m7.358603114s) [upload=446369]: Done uploading segments... Dec 26 20:10:07.487543534 (508h41m7.358626569s) [upload=446369]: Upload finished. err: <nil> Duration: 30.003074282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XXXXX......X.X.X.X.X..............XX..X..X...........X] duration: 30.003838662s trace: https://jaeger.dev.storj.io/trace/f373c6b40871a01 FAIL (2025-12-26 20:10:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HQWYSUUVCL4JJXFKADF4VPJJM2HT65Y74MCWEXTNCPB2GHSZHUMQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: SXFFEBDDRJ4VKK3VOWLNFG2ECLTW5WAFEPERTBWPJZ7CWPLSH5UA) context deadline exceeded ======LOGS====== 20:09:34.352724100 (508h40m34.223807135s) [upload=446363]: Starting upload Dec 26 20:09:34.352801357 (508h40m34.223884412s) [upload=446363]: Done starting upload Dec 26 20:09:34.352950611 (508h40m34.224033646s) [upload=446363]: Uploading segments... Dec 26 20:09:34.353041304 (508h40m34.224124339s) [upload=446363]: Got next segment. Inline: false Dec 26 20:09:34.353098813 (508h40m34.224181848s) [upload=446363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:09:34.353106238 (508h40m34.224189273s) [upload=446363]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:09:34.600596457 (508h40m34.471679492s) [seg_pos={0 0}, upload=446363]: Begin upload segment... Dec 26 20:09:34.600650860 (508h40m34.471733896s) [seg_pos={0 0}, upload=446363]: Done begin upload segment. Dec 26 20:09:34.600695556 (508h40m34.471778591s) [noise=true , address=145.79.227.15:4874 , piece_id=4IPCZNR3RIGLSNRXPYR5HYDC4A3UHHM4FPI3XY3PLDSSVX3NZL6Q, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.600737115 (508h40m34.47182015s) [noise=true , address=145.79.227.45:4229 , piece_id=HBXEL3VVJ6YGUEIHMAYLGOLSJIH5ITGRDYHCZH6M7I6XWJNYYJGA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.601287343 (508h40m34.472370378s) [noise=true , address=145.79.227.29:4942 , piece_id=LSNTFRY5GCEU4TAZBJSG7L6JLRAEDOILOVX25VUXL2PRI3XWSQIQ, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.601639083 (508h40m34.472722128s) [noise=true , address=145.79.227.48:4301 , piece_id=7JMGDOVSV6HO4WMEMKA4HIGTHBQD2R746ZDPXKLBJMXAFR4Y552Q, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.602190293 (508h40m34.473273338s) [noise=true , address=145.79.227.41:4085 , piece_id=XMQ32IW2M55Z3Z242QRVA2RNKMM4S4I2WRYKWBPOO5YVB3Z2VVPA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.602739357 (508h40m34.473822402s) [noise=true , address=145.79.227.42:4122 , piece_id=63VAN46P676K4WNKUDPAMZBPKTLMXXZVYOFSV3ADTF43THURJCZQ, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.603225613 (508h40m34.474308648s) [noise=true , address=145.79.227.50:4375 , piece_id=67SZQX3QTXNKGRELFQSHTSMYLGSW3ZOPUWSG44B7ZRCCSEWJBKHA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.604504849 (508h40m34.475587894s) [noise=true , address=45.91.116.9:4261 , piece_id=NCSDJARBSDSVXALFJRAMMFPWRGQRZOFEM2CENRZD5JGW7OPZXSYA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.604834999 (508h40m34.475918034s) [noise=true , address=145.79.227.36:4519 , piece_id=G3PWIJSERK5ZMR7ODR5BHOD33T6ZBKZ2Y4W2RWGZOGWCLNYU2TVA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.605992412 (508h40m34.477075447s) [noise=true , address=145.79.227.47:4265 , piece_id=XSLHQN3SZW3IG7EM2ECTIVE3O2CKUPFPXXMAJQOISNWEMHWXSMCA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.606463869 (508h40m34.477546905s) [noise=true , address=145.79.227.49:4337 , piece_id=2SHU7LNCEEEIKUQ2GM4573QQ4A5BUPLXG3P6RC4MZP4EF4QM4CPQ, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.607788121 (508h40m34.478871166s) [noise=true , address=77.90.31.13:4627 , piece_id=HQWYSUUVCL4JJXFKADF4VPJJM2HT65Y74MCWEXTNCPB2GHSZHUMQ, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.608331356 (508h40m34.479414391s) [noise=true , address=77.90.31.10:4517 , piece_id=4WUU3O23T3ZFXVJCNURTCEYPIQAXGH7YAWCOIV3VRXH5PBCMUD3A, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.609691285 (508h40m34.48077433s) [noise=true , address=45.91.116.12:4228 , piece_id=WR2UH6EK5HBO3662JWE6LGUV6SE6Q2S3MYYVZCCFS7DMSMPPXHKA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.611429114 (508h40m34.48251215s) [noise=true , address=45.91.116.11:4118 , piece_id=SXFFEBDDRJ4VKK3VOWLNFG2ECLTW5WAFEPERTBWPJZ7CWPLSH5UA, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:34.612744869 (508h40m34.483827904s) [noise=true , address=145.79.227.30:4762 , piece_id=IHTSKL2INKWI3VJZTUUXDSE4O42AB2WMRUETFSMCRJLB2D7TQZ4A, seg_pos={0 0}, upload=446363]: Uploading piece... Dec 26 20:09:36.231647050 (508h40m36.102730085s) [upload=446363]: Next returned nil segment Dec 26 20:09:36.231660084 (508h40m36.102743129s) [upload=446363]: Waiting for error group managing segments... Dec 26 20:09:36.615418930 (508h40m36.486501975s) [noise=true , address=145.79.227.42:4122 , piece_id=63VAN46P676K4WNKUDPAMZBPKTLMXXZVYOFSV3ADTF43THURJCZQ, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.617326583 (508h40m36.488409618s) [noise=true , address=145.79.227.45:4229 , piece_id=HBXEL3VVJ6YGUEIHMAYLGOLSJIH5ITGRDYHCZH6M7I6XWJNYYJGA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.619877290 (508h40m36.490960325s) [noise=true , address=145.79.227.15:4874 , piece_id=4IPCZNR3RIGLSNRXPYR5HYDC4A3UHHM4FPI3XY3PLDSSVX3NZL6Q, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.620003741 (508h40m36.491086786s) [noise=true , address=145.79.227.41:4085 , piece_id=XMQ32IW2M55Z3Z242QRVA2RNKMM4S4I2WRYKWBPOO5YVB3Z2VVPA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.620436555 (508h40m36.49151959s) [noise=true , address=145.79.227.50:4375 , piece_id=67SZQX3QTXNKGRELFQSHTSMYLGSW3ZOPUWSG44B7ZRCCSEWJBKHA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.625098572 (508h40m36.496181617s) [noise=true , address=145.79.227.29:4942 , piece_id=LSNTFRY5GCEU4TAZBJSG7L6JLRAEDOILOVX25VUXL2PRI3XWSQIQ, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.633402975 (508h40m36.50448602s) [noise=true , address=145.79.227.47:4265 , piece_id=XSLHQN3SZW3IG7EM2ECTIVE3O2CKUPFPXXMAJQOISNWEMHWXSMCA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.634255558 (508h40m36.505338593s) [noise=true , address=145.79.227.49:4337 , piece_id=2SHU7LNCEEEIKUQ2GM4573QQ4A5BUPLXG3P6RC4MZP4EF4QM4CPQ, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.641511535 (508h40m36.51259458s) [noise=true , address=145.79.227.36:4519 , piece_id=G3PWIJSERK5ZMR7ODR5BHOD33T6ZBKZ2Y4W2RWGZOGWCLNYU2TVA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.864288937 (508h40m36.735371972s) [noise=true , address=77.90.31.13:4627 , piece_id=HQWYSUUVCL4JJXFKADF4VPJJM2HT65Y74MCWEXTNCPB2GHSZHUMQ, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.913045898 (508h40m36.784128953s) [noise=true , address=45.91.116.12:4228 , piece_id=WR2UH6EK5HBO3662JWE6LGUV6SE6Q2S3MYYVZCCFS7DMSMPPXHKA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.930955932 (508h40m36.802038977s) [noise=true , address=77.90.31.10:4517 , piece_id=4WUU3O23T3ZFXVJCNURTCEYPIQAXGH7YAWCOIV3VRXH5PBCMUD3A, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.967034261 (508h40m36.838117306s) [noise=true , address=45.91.116.11:4118 , piece_id=SXFFEBDDRJ4VKK3VOWLNFG2ECLTW5WAFEPERTBWPJZ7CWPLSH5UA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: <nil> Dec 26 20:09:36.967056493 (508h40m36.838139538s) [seg_pos={0 0}, upload=446363]: Segment reached optimal threshold of 13 pieces. Dec 26 20:09:36.967291710 (508h40m36.838374765s) [noise=true , address=145.79.227.30:4762 , piece_id=IHTSKL2INKWI3VJZTUUXDSE4O42AB2WMRUETFSMCRJLB2D7TQZ4A, seg_pos={0 0}, upload=446363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 20:09:36.967304706 (508h40m36.838387751s) [noise=true , address=145.79.227.30:4762 , piece_id=IHTSKL2INKWI3VJZTUUXDSE4O42AB2WMRUETFSMCRJLB2D7TQZ4A, seg_pos={0 0}, upload=446363]: Long tail cancellation due to optimal threshold Dec 26 20:09:36.967476353 (508h40m36.838559398s) [noise=true , address=145.79.227.48:4301 , piece_id=7JMGDOVSV6HO4WMEMKA4HIGTHBQD2R746ZDPXKLBJMXAFR4Y552Q, seg_pos={0 0}, upload=446363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:52906->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:52906->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52906->145.79.227.48:4301: use of closed network connection Dec 26 20:09:36.967486121 (508h40m36.838569166s) [noise=true , address=145.79.227.48:4301 , piece_id=7JMGDOVSV6HO4WMEMKA4HIGTHBQD2R746ZDPXKLBJMXAFR4Y552Q, seg_pos={0 0}, upload=446363]: Long tail cancellation due to optimal threshold Dec 26 20:09:36.967552037 (508h40m36.838635072s) [noise=true , address=45.91.116.9:4261 , piece_id=NCSDJARBSDSVXALFJRAMMFPWRGQRZOFEM2CENRZD5JGW7OPZXSYA, seg_pos={0 0}, upload=446363]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:09:36.967563829 (508h40m36.838646864s) [noise=true , address=45.91.116.9:4261 , piece_id=NCSDJARBSDSVXALFJRAMMFPWRGQRZOFEM2CENRZD5JGW7OPZXSYA, seg_pos={0 0}, upload=446363]: Long tail cancellation due to optimal threshold Dec 26 20:09:36.967584729 (508h40m36.838667764s) [seg_pos={0 0}, upload=446363]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:09:36.967675982 (508h40m36.838759017s) [upload=446363]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:09:36.967683177 (508h40m36.838766222s) [upload=446363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:09:37.256841338 (508h40m37.127924373s) [upload=446363]: Done uploading segments... Dec 26 20:09:37.256863631 (508h40m37.127946666s) [upload=446363]: Upload finished. err: <nil> Duration: 30.003859642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 37.162833957s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 37.162942644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 26.429412312s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:09:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 26.429439163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 12.153803017s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 12.153834086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X.XXXXX......X.X.X.X.X..............XX..X..X...........] duration: 7.155259398s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:08:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 7.155287571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 19.213005975s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:07:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 19.213036322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X.XXXXX......X.X.X.X.X..............XX..X..X..........] duration: 6.18247538s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:07:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.182504324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 30.751414966s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:06:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 30.751440553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X....X.XXXXX......X.X.X.X.X..............XX..X..X.........] duration: 7.869037903s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:06:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 7.869182059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 13.248959428s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:05:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 13.248986087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.X....X.XXXXX......X.X.X.X.X..............XX..X..X........] duration: 6.256454625s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:05:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.256481465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.461814546s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:05:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 15.461850426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X.......] duration: 6.119082965s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:04:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.119109395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 30.743724415s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 30.743756605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X......] duration: 6.766421259s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:03:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.766446636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 11.732573303s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:03:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 11.732645812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X.....] duration: 8.024529154s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:02:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 8.02455782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X....] duration: 17.060677209s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:02:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 17.060702226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 20.098617013s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 20.098633124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 47.908034616s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 47.90805283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 34.262538682s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 34.262561364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X...] duration: 7.2398884s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:01:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 7.239939849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X..] duration: 6.181924667s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:00:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.181952991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 24.136007023s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:00:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 24.136032962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X.] duration: 6.26225004s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 20:00:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 6.262281902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..X] duration: 30.003091249s trace: https://jaeger.dev.storj.io/trace/31ce310036d1aba8 FAIL (2025-12-26 19:59:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 19:59:25.543208365 (508h30m25.4142914s) [upload=446219]: Starting upload Dec 26 19:59:25.543265393 (508h30m25.414348428s) [upload=446219]: Done starting upload Dec 26 19:59:25.543622093 (508h30m25.414705128s) [upload=446219]: Uploading segments... Dec 26 19:59:25.543964935 (508h30m25.41504797s) [upload=446219]: Got next segment. Inline: false Dec 26 19:59:25.544048475 (508h30m25.41513152s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:59:25.544055398 (508h30m25.415138443s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:59:25.803360680 (508h30m25.674443725s) [seg_pos={0 0}, upload=446219]: Begin upload segment... Dec 26 19:59:25.803443648 (508h30m25.674526693s) [seg_pos={0 0}, upload=446219]: Done begin upload segment. Dec 26 19:59:25.803469237 (508h30m25.674552282s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.808790760 (508h30m25.679873805s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.811220988 (508h30m25.682304033s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.813651294 (508h30m25.68473434s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.815518380 (508h30m25.686601425s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.817013828 (508h30m25.688096893s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.819450848 (508h30m25.690533893s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.823668349 (508h30m25.694751384s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.828779932 (508h30m25.699862977s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.832704766 (508h30m25.703787811s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.834700135 (508h30m25.70578317s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.835201220 (508h30m25.706284255s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.838266928 (508h30m25.709349973s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.840142588 (508h30m25.711225633s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.841545250 (508h30m25.712628295s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:25.843076867 (508h30m25.714159912s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Uploading piece... Dec 26 19:59:27.557472934 (508h30m27.428555969s) [upload=446219]: Next returned nil segment Dec 26 19:59:27.557486660 (508h30m27.428569705s) [upload=446219]: Waiting for error group managing segments... Dec 26 19:59:27.839878373 (508h30m27.710961408s) [noise=true , address=145.79.227.8:4330 , piece_id=DA2Y2DUMG4C3KAGSNUUSDBK6LJU5MFFMQHQZLZPG43UTUWXMNUZA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852387573 (508h30m27.723470608s) [noise=true , address=145.79.227.52:4447 , piece_id=CQ4CHKIGQWYKQXP7DLTPF3T6RLTH44SZMX5AJXPSZRZTKYFZDVBA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.852583315 (508h30m27.72366635s) [noise=true , address=145.79.227.38:4977 , piece_id=IBTSAMB4CU7HFCWDEUSWEW6TVJZNWSMXWC34RED42UI73ZW6AZJQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.855435376 (508h30m27.726518411s) [noise=true , address=145.79.227.21:4475 , piece_id=ZGGRM4CCTM7IHKOY5Z3XNPHUTZWSTO2D7EPW22E3MYIZ73STGPQQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.865759525 (508h30m27.73684256s) [noise=true , address=145.79.227.48:4302 , piece_id=OSSI7ZTGOO74HPUYGYS7YQ5YHBLLGXRQ5H36OCM2Z2U2JKHXA4TA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.878158886 (508h30m27.749241921s) [noise=true , address=145.79.227.46:4603 , piece_id=7S5XTTDC6D4NANGFP3HFN45C6ATNA6BPXQ4NEEL45UW2JS53J6XA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:27.886042767 (508h30m27.757125813s) [noise=true , address=145.79.227.16:4912 , piece_id=2L4R735XGK2PI7SWT3BLTGDZUANQO4M27HBYADOW6H6OZIRXMMMA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.055319009 (508h30m27.926402044s) [noise=true , address=45.91.116.13:4156 , piece_id=ZOFOLY2FQ7DUGPU3ESB7XI4SPUQCHV2ULGBIG73HGKRWTGGEOCZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.110737796 (508h30m27.981820831s) [noise=true , address=45.91.116.11:4120 , piece_id=WFQEFPBN67AHCSX3YP27RE7QEJM6K32XV6JKUY67G235QR3GOQFQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.136181325 (508h30m28.00726436s) [noise=true , address=45.91.116.9:4263 , piece_id=SWCBPNOG2OL3DKJ33RKGZH76YVEN63DQ75SSAXUMAFZAM7W6WVQA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.137659710 (508h30m28.008742745s) [noise=true , address=45.91.116.12:4225 , piece_id=5DHX22K3GIK3AECBPCLIZDXWDK7N4A6PEPJJAB5DTXAWV2RGPCWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.252929584 (508h30m28.124012619s) [noise=true , address=77.90.31.13:4627 , piece_id=FTTNQC5BHG4JGTKTL4FBR2IQQRT4YKMB3MNR7QNEIIUKH2PXVXZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702222485 (508h30m28.57330553s) [noise=true , address=77.90.31.14:4663 , piece_id=W65FEGE26SA7DNTJUHJUQYULCDH53FPTCUX33Q7RZCPAYW663PUA, seg_pos={0 0}, upload=446219]: Done uploading piece. err: <nil> Dec 26 19:59:28.702246501 (508h30m28.573329546s) [seg_pos={0 0}, upload=446219]: Segment reached optimal threshold of 13 pieces. Dec 26 19:59:28.702483914 (508h30m28.573566949s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702493782 (508h30m28.573576827s) [noise=true , address=145.79.227.33:4870 , piece_id=35T34GS5PH3A2K4MXKWPGPNA5HPSOMM3B5ZJR2EJK4BFHWYNCIWQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702627627 (508h30m28.573710662s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:59:28.702643968 (508h30m28.573727003s) [noise=true , address=77.90.31.16:4735 , piece_id=L4URTZ4DMLSCDV5YKYTCAP42QIXKQQXT5TRRJEVQGSRLI4MO4ZZQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702832267 (508h30m28.573915302s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.5:4298: use of closed network connection Dec 26 19:59:28.702838970 (508h30m28.573922005s) [noise=true , address=45.91.116.5:4298 , piece_id=3TURUMUOTZXB3MGVGIGEM7PC3PD6OA5FJKYZQQPZIQCM5JKUG7IQ, seg_pos={0 0}, upload=446219]: Long tail cancellation due to optimal threshold Dec 26 19:59:28.702860541 (508h30m28.573943586s) [seg_pos={0 0}, upload=446219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:59:28.702950482 (508h30m28.574033517s) [upload=446219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:59:28.702957274 (508h30m28.574040309s) [upload=446219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:59:29.000776925 (508h30m28.87185999s) [upload=446219]: Done uploading segments... Dec 26 19:59:29.000801452 (508h30m28.871884497s) [upload=446219]: Upload finished. err: <nil> Duration: 30.003114013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 24.88777004s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:59:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 24.887796189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X..] duration: 6.996955087s trace: https://jaeger.dev.storj.io/trace/14d4b37c8e59b5d9 FAIL (2025-12-26 19:58:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ) context deadline exceeded ======LOGS====== Dec 26 19:57:17.027220709 (508h28m16.898303744s) [upload=446185]: Starting upload Dec 26 19:57:17.027283498 (508h28m16.898366533s) [upload=446185]: Done starting upload Dec 26 19:57:17.027311291 (508h28m16.898394326s) [upload=446185]: Uploading segments... Dec 26 19:57:17.027487286 (508h28m16.898570332s) [upload=446185]: Got next segment. Inline: false Dec 26 19:57:17.027518807 (508h28m16.898601842s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:57:17.027523004 (508h28m16.89860604s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:57:17.273656535 (508h28m17.14473958s) [seg_pos={0 0}, upload=446185]: Begin upload segment... Dec 26 19:57:17.273714515 (508h28m17.14479756s) [seg_pos={0 0}, upload=446185]: Done begin upload segment. Dec 26 19:57:17.273745284 (508h28m17.144828319s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.273756445 (508h28m17.14483949s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274236279 (508h28m17.145319324s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274391414 (508h28m17.145474459s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274681727 (508h28m17.145764772s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274714890 (508h28m17.145797926s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275047403 (508h28m17.146130448s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275426235 (508h28m17.14650928s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275787503 (508h28m17.146870538s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275861554 (508h28m17.146944589s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276155263 (508h28m17.147238298s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276424456 (508h28m17.147507491s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276757129 (508h28m17.147840164s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276843944 (508h28m17.147926979s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277240259 (508h28m17.148323294s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277674286 (508h28m17.148757321s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:18.939539535 (508h28m18.81062258s) [upload=446185]: Next returned nil segment Dec 26 19:57:18.939564262 (508h28m18.810647307s) [upload=446185]: Waiting for error group managing segments... Dec 26 19:57:19.263812608 (508h28m19.134895643s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.272414407 (508h28m19.143497442s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.273393551 (508h28m19.144476586s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.274322921 (508h28m19.145405966s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.279223242 (508h28m19.150306287s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284611362 (508h28m19.155694397s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284982048 (508h28m19.156065083s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.285753416 (508h28m19.156836461s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.296923206 (508h28m19.168006241s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.297194062 (508h28m19.168277107s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.306966961 (508h28m19.178050006s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.509409126 (508h28m19.380492171s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510608549 (508h28m19.381691594s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510621935 (508h28m19.38170498s) [seg_pos={0 0}, upload=446185]: Segment reached optimal threshold of 13 pieces. Dec 26 19:57:19.510842765 (508h28m19.38192581s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510853496 (508h28m19.381936531s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.510929822 (508h28m19.382012857s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510941373 (508h28m19.382024408s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511108592 (508h28m19.382191627s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection Dec 26 19:57:19.511115024 (508h28m19.382198059s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511130743 (508h28m19.382213778s) [seg_pos={0 0}, upload=446185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:57:19.511207180 (508h28m19.382290225s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:57:19.511213071 (508h28m19.382296116s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:57:19.826403868 (508h28m19.697486913s) [upload=446185]: Done uploading segments... Dec 26 19:57:19.826425349 (508h28m19.697508394s) [upload=446185]: Upload finished. err: <nil> Duration: 6.996970526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X.] duration: 18.650958261s trace: https://jaeger.dev.storj.io/trace/14d4b37c8e59b5d9 FAIL (2025-12-26 19:58:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ) context deadline exceeded ======LOGS====== Dec 26 19:57:17.027220709 (508h28m16.898303744s) [upload=446185]: Starting upload Dec 26 19:57:17.027283498 (508h28m16.898366533s) [upload=446185]: Done starting upload Dec 26 19:57:17.027311291 (508h28m16.898394326s) [upload=446185]: Uploading segments... Dec 26 19:57:17.027487286 (508h28m16.898570332s) [upload=446185]: Got next segment. Inline: false Dec 26 19:57:17.027518807 (508h28m16.898601842s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:57:17.027523004 (508h28m16.89860604s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:57:17.273656535 (508h28m17.14473958s) [seg_pos={0 0}, upload=446185]: Begin upload segment... Dec 26 19:57:17.273714515 (508h28m17.14479756s) [seg_pos={0 0}, upload=446185]: Done begin upload segment. Dec 26 19:57:17.273745284 (508h28m17.144828319s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.273756445 (508h28m17.14483949s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274236279 (508h28m17.145319324s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274391414 (508h28m17.145474459s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274681727 (508h28m17.145764772s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274714890 (508h28m17.145797926s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275047403 (508h28m17.146130448s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275426235 (508h28m17.14650928s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275787503 (508h28m17.146870538s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275861554 (508h28m17.146944589s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276155263 (508h28m17.147238298s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276424456 (508h28m17.147507491s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276757129 (508h28m17.147840164s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276843944 (508h28m17.147926979s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277240259 (508h28m17.148323294s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277674286 (508h28m17.148757321s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:18.939539535 (508h28m18.81062258s) [upload=446185]: Next returned nil segment Dec 26 19:57:18.939564262 (508h28m18.810647307s) [upload=446185]: Waiting for error group managing segments... Dec 26 19:57:19.263812608 (508h28m19.134895643s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.272414407 (508h28m19.143497442s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.273393551 (508h28m19.144476586s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.274322921 (508h28m19.145405966s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.279223242 (508h28m19.150306287s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284611362 (508h28m19.155694397s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284982048 (508h28m19.156065083s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.285753416 (508h28m19.156836461s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.296923206 (508h28m19.168006241s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.297194062 (508h28m19.168277107s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.306966961 (508h28m19.178050006s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.509409126 (508h28m19.380492171s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510608549 (508h28m19.381691594s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510621935 (508h28m19.38170498s) [seg_pos={0 0}, upload=446185]: Segment reached optimal threshold of 13 pieces. Dec 26 19:57:19.510842765 (508h28m19.38192581s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510853496 (508h28m19.381936531s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.510929822 (508h28m19.382012857s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510941373 (508h28m19.382024408s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511108592 (508h28m19.382191627s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection Dec 26 19:57:19.511115024 (508h28m19.382198059s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511130743 (508h28m19.382213778s) [seg_pos={0 0}, upload=446185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:57:19.511207180 (508h28m19.382290225s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:57:19.511213071 (508h28m19.382296116s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:57:19.826403868 (508h28m19.697486913s) [upload=446185]: Done uploading segments... Dec 26 19:57:19.826425349 (508h28m19.697508394s) [upload=446185]: Upload finished. err: <nil> Duration: 18.650987326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 14.580038691s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:58:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 14.5800581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..X] duration: 30.002510934s trace: https://jaeger.dev.storj.io/trace/14d4b37c8e59b5d9 FAIL (2025-12-26 19:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ) context deadline exceeded ======LOGS====== Dec 26 19:57:17.027220709 (508h28m16.898303744s) [upload=446185]: Starting upload Dec 26 19:57:17.027283498 (508h28m16.898366533s) [upload=446185]: Done starting upload Dec 26 19:57:17.027311291 (508h28m16.898394326s) [upload=446185]: Uploading segments... Dec 26 19:57:17.027487286 (508h28m16.898570332s) [upload=446185]: Got next segment. Inline: false Dec 26 19:57:17.027518807 (508h28m16.898601842s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:57:17.027523004 (508h28m16.89860604s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:57:17.273656535 (508h28m17.14473958s) [seg_pos={0 0}, upload=446185]: Begin upload segment... Dec 26 19:57:17.273714515 (508h28m17.14479756s) [seg_pos={0 0}, upload=446185]: Done begin upload segment. Dec 26 19:57:17.273745284 (508h28m17.144828319s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.273756445 (508h28m17.14483949s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274236279 (508h28m17.145319324s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274391414 (508h28m17.145474459s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274681727 (508h28m17.145764772s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.274714890 (508h28m17.145797926s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275047403 (508h28m17.146130448s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275426235 (508h28m17.14650928s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275787503 (508h28m17.146870538s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.275861554 (508h28m17.146944589s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276155263 (508h28m17.147238298s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276424456 (508h28m17.147507491s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276757129 (508h28m17.147840164s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.276843944 (508h28m17.147926979s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277240259 (508h28m17.148323294s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:17.277674286 (508h28m17.148757321s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Uploading piece... Dec 26 19:57:18.939539535 (508h28m18.81062258s) [upload=446185]: Next returned nil segment Dec 26 19:57:18.939564262 (508h28m18.810647307s) [upload=446185]: Waiting for error group managing segments... Dec 26 19:57:19.263812608 (508h28m19.134895643s) [noise=true , address=145.79.227.52:4447 , piece_id=HQQFDBIAXIQ7ZZ3JH37OU6SPBQ5ECKIVYM2I5P7OAYLJSRNTGLLQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.272414407 (508h28m19.143497442s) [noise=true , address=145.79.227.22:4509 , piece_id=R3N6P4WHTGP6GG7OGAZPRBOLJTDH2EM7GW4LANBDJUL6LSM7F52A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.273393551 (508h28m19.144476586s) [noise=true , address=145.79.227.50:4375 , piece_id=OILZBTNZ2EXAXQATPBWKOTQGS7YQCUN5BGLL3CDAZSRUB4AALM6Q, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.274322921 (508h28m19.145405966s) [noise=true , address=145.79.227.39:4014 , piece_id=IY3ZJ4LU52ZJUCD5PXG7RRZ6ZHVH3DANZIVCOURO3Y4KSQNY7GPA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.279223242 (508h28m19.150306287s) [noise=true , address=145.79.227.38:4979 , piece_id=3LAREWM7QLRMHNHBUSNSVKNS7GQE4F3T7QIH5XOPVAYCSEXKJV6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284611362 (508h28m19.155694397s) [noise=true , address=145.79.227.15:4874 , piece_id=MC7Q4Z44J2S2DG7PI5FDNJLRWQDXPIO7EYWYMPP6F2OOFPKCHVWQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.284982048 (508h28m19.156065083s) [noise=true , address=145.79.227.31:4799 , piece_id=3IJ6UDUWRUH3URROE63BXOW7CRULXVGTM53OHBPGOKOS2ZEOVR6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.285753416 (508h28m19.156836461s) [noise=true , address=145.79.227.48:4303 , piece_id=HZG4RYTGUGWC6N35RZPWHLCISI5UNTNV2ILDQ5BIU2AVCQLQNSJA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.296923206 (508h28m19.168006241s) [noise=true , address=145.79.227.49:4339 , piece_id=IM5T65A4WG6JFZZ3OO2DQDNDHWRZKDSTC5XICC4JTBHKOD4ZAV7A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.297194062 (508h28m19.168277107s) [noise=true , address=145.79.227.45:4231 , piece_id=HSCWVYTQB32N4FP6Q7EA5RUF3OOLF3XPILJ25XCLGSKNNX5WFITQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.306966961 (508h28m19.178050006s) [noise=true , address=145.79.227.41:4086 , piece_id=EUHYNO6LSFWC5FH2WVASKICJQRAYU6H4UYQ36QMRQOKFOJOKSAZA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.509409126 (508h28m19.380492171s) [noise=true , address=45.91.116.12:4225 , piece_id=2LYUYNF6CMTFMHCQNV42LV27SM4QK4LDWPBS4W5DQBJV5OUXPAWA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510608549 (508h28m19.381691594s) [noise=true , address=45.91.116.5:4300 , piece_id=AZUVXZO6OYA537YRHMGSQIIFIBOS3KQDIOXEXVIZ3URZMBLEXPHQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: <nil> Dec 26 19:57:19.510621935 (508h28m19.38170498s) [seg_pos={0 0}, upload=446185]: Segment reached optimal threshold of 13 pieces. Dec 26 19:57:19.510842765 (508h28m19.38192581s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510853496 (508h28m19.381936531s) [noise=true , address=45.91.116.13:4156 , piece_id=CES44PABMSA7RPK32KXHOPGMASTPOFGTKQCEIF2VKAEDD4QNAU6A, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.510929822 (508h28m19.382012857s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 19:57:19.510941373 (508h28m19.382024408s) [noise=true , address=45.91.116.11:4119 , piece_id=LB3GNWTXQKD2GOEES5VBPGXB2OBL46XN2XKDQBDUUELUKZHYLRQA, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511108592 (508h28m19.382191627s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51528->45.91.116.4:4937: use of closed network connection Dec 26 19:57:19.511115024 (508h28m19.382198059s) [noise=true , address=45.91.116.4:4937 , piece_id=KOF2HLCZASX6DU7IFVTCYWQWRBYXIGHJR5AFHDV4T6NKKB45UWSQ, seg_pos={0 0}, upload=446185]: Long tail cancellation due to optimal threshold Dec 26 19:57:19.511130743 (508h28m19.382213778s) [seg_pos={0 0}, upload=446185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:57:19.511207180 (508h28m19.382290225s) [upload=446185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:57:19.511213071 (508h28m19.382296116s) [upload=446185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:57:19.826403868 (508h28m19.697486913s) [upload=446185]: Done uploading segments... Dec 26 19:57:19.826425349 (508h28m19.697508394s) [upload=446185]: Upload finished. err: <nil> Duration: 30.00253505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.367295322s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:57:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 8.367311472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 18.082255349s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 18.082279735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX..] duration: 5.939968851s trace: https://jaeger.dev.storj.io/trace/582dd59baac1ab33 FAIL (2025-12-26 19:57:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA) context deadline exceeded ======LOGS====== nts... Dec 26 19:54:54.255492719 (508h25m54.126575754s) [upload=446154]: Got next segment. Inline: false Dec 26 19:54:54.255573413 (508h25m54.126656458s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:54:54.255582170 (508h25m54.126665215s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:54:54.503709247 (508h25m54.374792322s) [seg_pos={0 0}, upload=446154]: Begin upload segment... Dec 26 19:54:54.503774932 (508h25m54.374857967s) [seg_pos={0 0}, upload=446154]: Done begin upload segment. Dec 26 19:54:54.503846468 (508h25m54.374929504s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.503855065 (508h25m54.3749381s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.504448665 (508h25m54.3755317s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505016486 (508h25m54.376099521s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505622049 (508h25m54.376705095s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505927972 (508h25m54.377011017s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.506408907 (508h25m54.377491952s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.510132147 (508h25m54.381215192s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.511599562 (508h25m54.382682597s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514397189 (508h25m54.385480224s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514941435 (508h25m54.386024471s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.515757920 (508h25m54.386840955s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.516223837 (508h25m54.387306882s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517313091 (508h25m54.388396126s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517893065 (508h25m54.38897611s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.518785585 (508h25m54.38986863s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:56.211864632 (508h25m56.082947667s) [upload=446154]: Next returned nil segment Dec 26 19:54:56.211883888 (508h25m56.082966953s) [upload=446154]: Waiting for error group managing segments... Dec 26 19:54:56.504235316 (508h25m56.375318362s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.506253119 (508h25m56.377336154s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509770617 (508h25m56.380853652s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509974986 (508h25m56.381058021s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.513031346 (508h25m56.384114381s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.521057939 (508h25m56.392140974s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.523412233 (508h25m56.394495278s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.541169265 (508h25m56.4122523s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.702442357 (508h25m56.573525392s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.713013487 (508h25m56.584096522s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.773849779 (508h25m56.644932814s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.794635709 (508h25m56.665718744s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863416038 (508h25m56.734499084s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863436548 (508h25m56.734519583s) [seg_pos={0 0}, upload=446154]: Segment reached optimal threshold of 13 pieces. Dec 26 19:54:56.863627902 (508h25m56.734710937s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 19:54:56.863634915 (508h25m56.73471795s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863807063 (508h25m56.734890098s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection Dec 26 19:54:56.863812985 (508h25m56.73489602s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863821250 (508h25m56.734904285s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection Dec 26 19:54:56.863825779 (508h25m56.734908824s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863838563 (508h25m56.734921598s) [seg_pos={0 0}, upload=446154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:54:56.863903206 (508h25m56.734986241s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:54:56.863908145 (508h25m56.73499118s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:54:57.151503487 (508h25m57.022586522s) [upload=446154]: Done uploading segments... Dec 26 19:54:57.151522283 (508h25m57.022605318s) [upload=446154]: Upload finished. err: <nil> Duration: 5.939992816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.353037796s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:56:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 21.353064197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX.] duration: 6.070222275s trace: https://jaeger.dev.storj.io/trace/582dd59baac1ab33 FAIL (2025-12-26 19:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA) context deadline exceeded ======LOGS====== nts... Dec 26 19:54:54.255492719 (508h25m54.126575754s) [upload=446154]: Got next segment. Inline: false Dec 26 19:54:54.255573413 (508h25m54.126656458s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:54:54.255582170 (508h25m54.126665215s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:54:54.503709247 (508h25m54.374792322s) [seg_pos={0 0}, upload=446154]: Begin upload segment... Dec 26 19:54:54.503774932 (508h25m54.374857967s) [seg_pos={0 0}, upload=446154]: Done begin upload segment. Dec 26 19:54:54.503846468 (508h25m54.374929504s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.503855065 (508h25m54.3749381s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.504448665 (508h25m54.3755317s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505016486 (508h25m54.376099521s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505622049 (508h25m54.376705095s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505927972 (508h25m54.377011017s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.506408907 (508h25m54.377491952s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.510132147 (508h25m54.381215192s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.511599562 (508h25m54.382682597s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514397189 (508h25m54.385480224s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514941435 (508h25m54.386024471s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.515757920 (508h25m54.386840955s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.516223837 (508h25m54.387306882s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517313091 (508h25m54.388396126s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517893065 (508h25m54.38897611s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.518785585 (508h25m54.38986863s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:56.211864632 (508h25m56.082947667s) [upload=446154]: Next returned nil segment Dec 26 19:54:56.211883888 (508h25m56.082966953s) [upload=446154]: Waiting for error group managing segments... Dec 26 19:54:56.504235316 (508h25m56.375318362s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.506253119 (508h25m56.377336154s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509770617 (508h25m56.380853652s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509974986 (508h25m56.381058021s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.513031346 (508h25m56.384114381s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.521057939 (508h25m56.392140974s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.523412233 (508h25m56.394495278s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.541169265 (508h25m56.4122523s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.702442357 (508h25m56.573525392s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.713013487 (508h25m56.584096522s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.773849779 (508h25m56.644932814s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.794635709 (508h25m56.665718744s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863416038 (508h25m56.734499084s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863436548 (508h25m56.734519583s) [seg_pos={0 0}, upload=446154]: Segment reached optimal threshold of 13 pieces. Dec 26 19:54:56.863627902 (508h25m56.734710937s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 19:54:56.863634915 (508h25m56.73471795s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863807063 (508h25m56.734890098s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection Dec 26 19:54:56.863812985 (508h25m56.73489602s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863821250 (508h25m56.734904285s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection Dec 26 19:54:56.863825779 (508h25m56.734908824s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863838563 (508h25m56.734921598s) [seg_pos={0 0}, upload=446154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:54:56.863903206 (508h25m56.734986241s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:54:56.863908145 (508h25m56.73499118s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:54:57.151503487 (508h25m57.022586522s) [upload=446154]: Done uploading segments... Dec 26 19:54:57.151522283 (508h25m57.022605318s) [upload=446154]: Upload finished. err: <nil> Duration: 6.070249538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............XX] duration: 30.00270453s trace: https://jaeger.dev.storj.io/trace/582dd59baac1ab33 FAIL (2025-12-26 19:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA) context deadline exceeded ======LOGS====== nts... Dec 26 19:54:54.255492719 (508h25m54.126575754s) [upload=446154]: Got next segment. Inline: false Dec 26 19:54:54.255573413 (508h25m54.126656458s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:54:54.255582170 (508h25m54.126665215s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:54:54.503709247 (508h25m54.374792322s) [seg_pos={0 0}, upload=446154]: Begin upload segment... Dec 26 19:54:54.503774932 (508h25m54.374857967s) [seg_pos={0 0}, upload=446154]: Done begin upload segment. Dec 26 19:54:54.503846468 (508h25m54.374929504s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.503855065 (508h25m54.3749381s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.504448665 (508h25m54.3755317s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505016486 (508h25m54.376099521s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505622049 (508h25m54.376705095s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.505927972 (508h25m54.377011017s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.506408907 (508h25m54.377491952s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.510132147 (508h25m54.381215192s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.511599562 (508h25m54.382682597s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514397189 (508h25m54.385480224s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.514941435 (508h25m54.386024471s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.515757920 (508h25m54.386840955s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.516223837 (508h25m54.387306882s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517313091 (508h25m54.388396126s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.517893065 (508h25m54.38897611s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:54.518785585 (508h25m54.38986863s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Uploading piece... Dec 26 19:54:56.211864632 (508h25m56.082947667s) [upload=446154]: Next returned nil segment Dec 26 19:54:56.211883888 (508h25m56.082966953s) [upload=446154]: Waiting for error group managing segments... Dec 26 19:54:56.504235316 (508h25m56.375318362s) [noise=true , address=145.79.227.48:4303 , piece_id=I3FKPQH7UPJREPB35QVXTAXSSCQKYH7DXGVIHQQKNYAWQEZGBMEA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.506253119 (508h25m56.377336154s) [noise=true , address=145.79.227.46:4601 , piece_id=FZHRL4XH6RD7L6XSGM5ZQXHNWC2VVAQ4GUD7ZNQVY2NEXKPCURHQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509770617 (508h25m56.380853652s) [noise=true , address=145.79.227.38:4977 , piece_id=XPAGBY5EGMDQM2V273IX3JOUIH53VUQSJW3BXJFR7RMJSO2UWX5Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.509974986 (508h25m56.381058021s) [noise=true , address=145.79.227.42:4123 , piece_id=NK3YPQPS54WXOPDASDTXJUKVWRV3CTHYY2M64NQM4TXYL2C3TNYA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.513031346 (508h25m56.384114381s) [noise=true , address=145.79.227.49:4339 , piece_id=JHHXZZ2I763FHZUX6UV6SIAAVDIDLOVFH2CUZA3HVNHL3KIL7S4Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.521057939 (508h25m56.392140974s) [noise=true , address=145.79.227.45:4230 , piece_id=EJ6BW5AFYQQ3R3QZBEBMHNXGJEUMWSIDIMWJGPS7S4E357BVPLTQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.523412233 (508h25m56.394495278s) [noise=true , address=145.79.227.23:4547 , piece_id=CAKX6WXCFWFANP3OM44CIBH7YTMX4LCJXNE5S6DKQHXYSHOLLT4A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.541169265 (508h25m56.4122523s) [noise=true , address=145.79.227.37:4566 , piece_id=ECHCENBSJA2AYJJBETQMAXXU7MJD6LFRNAITFF4IYAXMGN7IAYJA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.702442357 (508h25m56.573525392s) [noise=true , address=77.90.31.21:4880 , piece_id=M54PUY7WPOEVUKDTWUFSXLIUEOYGDPYVU45NJ74Q3DPKLJGUINIQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.713013487 (508h25m56.584096522s) [noise=true , address=77.90.31.14:4662 , piece_id=FC5KQN6XBS6SBEEN3ALRL3LOONOXFUMOQT2LC4WIJKLTWK34JQMA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.773849779 (508h25m56.644932814s) [noise=true , address=77.90.31.19:4844 , piece_id=2SPMX7FYSWW2VUC3RXF44FA5OHY5GPYDC26DIVHBFFAOYE5S7ULA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.794635709 (508h25m56.665718744s) [noise=true , address=45.91.116.13:4156 , piece_id=S4MAZCQP57DXSAERKTAPT6FYMOZ3YNOTWGEOJ2LIUUPQNLIQEFDA, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863416038 (508h25m56.734499084s) [noise=true , address=145.79.227.3:4150 , piece_id=TPI6DQEOJVA3ILEIL3HLCD4V6HK5DBZECEMORMEJYVFKE5PA5PRQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: <nil> Dec 26 19:54:56.863436548 (508h25m56.734519583s) [seg_pos={0 0}, upload=446154]: Segment reached optimal threshold of 13 pieces. Dec 26 19:54:56.863627902 (508h25m56.734710937s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 19:54:56.863634915 (508h25m56.73471795s) [noise=true , address=45.91.116.14:4192 , piece_id=NJRIDX7ZTINA57ACIVQ2R2LDJVU42DH4HCX4CY5GVBMCKVBU435A, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863807063 (508h25m56.734890098s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39318->145.79.227.19:4079: use of closed network connection Dec 26 19:54:56.863812985 (508h25m56.73489602s) [noise=true , address=145.79.227.19:4079 , piece_id=2JL5BBEWG6RP5PBGDKFEVYGYKVLTCAIXG6RYNXNNSLNNETXE6ZYQ, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863821250 (508h25m56.734904285s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60604->145.79.227.15:4876: use of closed network connection Dec 26 19:54:56.863825779 (508h25m56.734908824s) [noise=true , address=145.79.227.15:4876 , piece_id=6IFDMPTVDJFH4D6DUNJK57FJIW3RIY7TX3AU6KTPLKTZ2TL5PI2Q, seg_pos={0 0}, upload=446154]: Long tail cancellation due to optimal threshold Dec 26 19:54:56.863838563 (508h25m56.734921598s) [seg_pos={0 0}, upload=446154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:54:56.863903206 (508h25m56.734986241s) [upload=446154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:54:56.863908145 (508h25m56.73499118s) [upload=446154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:54:57.151503487 (508h25m57.022586522s) [upload=446154]: Done uploading segments... Dec 26 19:54:57.151522283 (508h25m57.022605318s) [upload=446154]: Upload finished. err: <nil> Duration: 30.002733265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 19.441324624s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:54:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 19.441338931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X..XXX.X...XXX.X....X.XXXXX......X.X.X.X.X..............X] duration: 30.005762864s trace: https://jaeger.dev.storj.io/trace/5da671104a394e35 FAIL (2025-12-26 19:54:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 19:53:46.638027314 (508h24m46.509110349s) [upload=446139]: Starting upload Dec 26 19:53:46.638086588 (508h24m46.509169623s) [upload=446139]: Done starting upload Dec 26 19:53:46.638250058 (508h24m46.509333093s) [upload=446139]: Uploading segments... Dec 26 19:53:46.638284474 (508h24m46.509367509s) [upload=446139]: Got next segment. Inline: false Dec 26 19:53:46.638307648 (508h24m46.509390693s) [upload=446139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 19:53:46.638311556 (508h24m46.509394591s) [upload=446139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 19:53:46.883848431 (508h24m46.754931476s) [seg_pos={0 0}, upload=446139]: Begin upload segment... Dec 26 19:53:46.883909227 (508h24m46.754992262s) [seg_pos={0 0}, upload=446139]: Done begin upload segment. Dec 26 19:53:46.883948301 (508h24m46.755031336s) [noise=true , address=45.91.116.12:4226 , piece_id=DYRQXVRHQDN4KGNEUENCN4EHYWQM3MX42AMXJLHYNJF3VDNYAWVQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.883967929 (508h24m46.755050984s) [noise=true , address=45.91.116.7:4011 , piece_id=EACSOAWB3FOLTKVL2P3DXCW576Z2TL2M3RSEJHJJCFQHR6GVCYXQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.884155586 (508h24m46.755238631s) [noise=true , address=77.90.31.16:4735 , piece_id=XRANRGJZKUQQNWBDN44BSCHFQCDL3L6CZJXCSTXWSKF37WLQY4LA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.884928527 (508h24m46.756011582s) [noise=true , address=45.91.116.14:4190 , piece_id=BHDNXXLCXGHN66QMV4HQJCF5KRSGPXKIEF57RK3ESF4SN3EHZ6TQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.885426205 (508h24m46.75650924s) [noise=true , address=45.91.116.6:4976 , piece_id=F7IPJEYC4B6JXBKCC3EISZMMJGK7JBFUH347V3NR4P3ESBHJAHCA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.885767665 (508h24m46.75685071s) [noise=true , address=145.79.227.38:4978 , piece_id=VV4BQCBYAIPIVLMDQ7BLUN7KOO4DLWARKO7U3WK2GQ3FR5XOEIGA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.886239544 (508h24m46.757322579s) [noise=true , address=145.79.227.39:4015 , piece_id=BL5MHBZ4YBQ6H76IIR3QZD3VVCC5F27XI7EIYSJAEHKUNRXBXAAA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.886844425 (508h24m46.757927471s) [noise=true , address=45.91.116.11:4119 , piece_id=DUXKDJMDR2QSFDFUJSWORDKES5XFAO3SON5J2XTLVMNSYUW6ZDVQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.887000203 (508h24m46.758083238s) [noise=true , address=45.91.116.13:4156 , piece_id=ANKS2BXRT5FBA5WFDYDZT46GEEE3D5JXMSGC2O5GFIU3CZMFNLIA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.887348706 (508h24m46.758431741s) [noise=true , address=45.91.116.8:4047 , piece_id=HYIB6SZLXZV32NWJIIRJ5THKKL7P2JGKGWUKAPVCAZCLCXR2I53Q, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.887547224 (508h24m46.758630269s) [noise=true , address=145.79.227.45:4231 , piece_id=WO2C6C75HMHWKHOXVY6MIJWFBGYFZM362NYRWK6XOQFQS5SL5EBA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.887986400 (508h24m46.759069445s) [noise=true , address=145.79.227.52:4447 , piece_id=4FSH745IVGYAHSYSHUSGPTA4JRVA7KBPPCTT32U5CP326AD62MDQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.888438711 (508h24m46.759521757s) [noise=true , address=145.79.227.37:4565 , piece_id=2H2EMUWUQLH26AERVYNY5OOSU7BEOVD6HLNBI5NRMIZZ5S6I6JQQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.889003797 (508h24m46.760086832s) [noise=true , address=45.91.116.9:4262 , piece_id=VDZ2EFA7SG3MEPBEKZ7GQVOQEW4KFGYMDYS6JEG574VZWNUNKYLA, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.889551480 (508h24m46.760634525s) [noise=true , address=145.79.227.19:4079 , piece_id=B7IJ7QUZMAFLA4DZOHYJMVMVZLUDCP4RPXCOLIXDMX5RRXV3VJKQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:46.889880046 (508h24m46.760963081s) [noise=true , address=45.91.116.5:4300 , piece_id=KYODECLSJQHCV6MSH7K6RH57BNJ77PQJWQZCH2FXLNCEVJRWE5LQ, seg_pos={0 0}, upload=446139]: Uploading piece... Dec 26 19:53:48.599539414 (508h24m48.470622459s) [upload=446139]: Next returned nil segment Dec 26 19:53:48.599553160 (508h24m48.470636195s) [upload=446139]: Waiting for error group managing segments... Dec 26 19:53:48.873036731 (508h24m48.744119766s) [noise=true , address=145.79.227.38:4978 , piece_id=VV4BQCBYAIPIVLMDQ7BLUN7KOO4DLWARKO7U3WK2GQ3FR5XOEIGA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:48.878016112 (508h24m48.749099157s) [noise=true , address=145.79.227.39:4015 , piece_id=BL5MHBZ4YBQ6H76IIR3QZD3VVCC5F27XI7EIYSJAEHKUNRXBXAAA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:48.887935800 (508h24m48.759018845s) [noise=true , address=145.79.227.45:4231 , piece_id=WO2C6C75HMHWKHOXVY6MIJWFBGYFZM362NYRWK6XOQFQS5SL5EBA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:48.892516513 (508h24m48.763599548s) [noise=true , address=145.79.227.19:4079 , piece_id=B7IJ7QUZMAFLA4DZOHYJMVMVZLUDCP4RPXCOLIXDMX5RRXV3VJKQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:48.902083300 (508h24m48.773166335s) [noise=true , address=145.79.227.37:4565 , piece_id=2H2EMUWUQLH26AERVYNY5OOSU7BEOVD6HLNBI5NRMIZZ5S6I6JQQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:48.914398721 (508h24m48.785481756s) [noise=true , address=145.79.227.52:4447 , piece_id=4FSH745IVGYAHSYSHUSGPTA4JRVA7KBPPCTT32U5CP326AD62MDQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.113245950 (508h24m48.984328985s) [noise=true , address=45.91.116.11:4119 , piece_id=DUXKDJMDR2QSFDFUJSWORDKES5XFAO3SON5J2XTLVMNSYUW6ZDVQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.169684229 (508h24m49.040767264s) [noise=true , address=45.91.116.5:4300 , piece_id=KYODECLSJQHCV6MSH7K6RH57BNJ77PQJWQZCH2FXLNCEVJRWE5LQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.181974542 (508h24m49.053057577s) [noise=true , address=45.91.116.14:4190 , piece_id=BHDNXXLCXGHN66QMV4HQJCF5KRSGPXKIEF57RK3ESF4SN3EHZ6TQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.189093186 (508h24m49.060176221s) [noise=true , address=45.91.116.7:4011 , piece_id=EACSOAWB3FOLTKVL2P3DXCW576Z2TL2M3RSEJHJJCFQHR6GVCYXQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.201422353 (508h24m49.072505388s) [noise=true , address=45.91.116.9:4262 , piece_id=VDZ2EFA7SG3MEPBEKZ7GQVOQEW4KFGYMDYS6JEG574VZWNUNKYLA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.235613369 (508h24m49.106696415s) [noise=true , address=45.91.116.13:4156 , piece_id=ANKS2BXRT5FBA5WFDYDZT46GEEE3D5JXMSGC2O5GFIU3CZMFNLIA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.244832104 (508h24m49.115915139s) [noise=true , address=45.91.116.8:4047 , piece_id=HYIB6SZLXZV32NWJIIRJ5THKKL7P2JGKGWUKAPVCAZCLCXR2I53Q, seg_pos={0 0}, upload=446139]: Done uploading piece. err: <nil> Dec 26 19:53:49.244856259 (508h24m49.115939294s) [seg_pos={0 0}, upload=446139]: Segment reached optimal threshold of 13 pieces. Dec 26 19:53:49.245046352 (508h24m49.116129387s) [noise=true , address=45.91.116.6:4976 , piece_id=F7IPJEYC4B6JXBKCC3EISZMMJGK7JBFUH347V3NR4P3ESBHJAHCA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38100->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38100->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38100->45.91.116.6:4976: use of closed network connection Dec 26 19:53:49.245052944 (508h24m49.116135989s) [noise=true , address=45.91.116.6:4976 , piece_id=F7IPJEYC4B6JXBKCC3EISZMMJGK7JBFUH347V3NR4P3ESBHJAHCA, seg_pos={0 0}, upload=446139]: Long tail cancellation due to optimal threshold Dec 26 19:53:49.245154728 (508h24m49.116237773s) [noise=true , address=45.91.116.12:4226 , piece_id=DYRQXVRHQDN4KGNEUENCN4EHYWQM3MX42AMXJLHYNJF3VDNYAWVQ, seg_pos={0 0}, upload=446139]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 19:53:49.245168013 (508h24m49.116251048s) [noise=true , address=45.91.116.12:4226 , piece_id=DYRQXVRHQDN4KGNEUENCN4EHYWQM3MX42AMXJLHYNJF3VDNYAWVQ, seg_pos={0 0}, upload=446139]: Long tail cancellation due to optimal threshold Dec 26 19:53:49.245189985 (508h24m49.11627302s) [noise=true , address=77.90.31.16:4735 , piece_id=XRANRGJZKUQQNWBDN44BSCHFQCDL3L6CZJXCSTXWSKF37WLQY4LA, seg_pos={0 0}, upload=446139]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 19:53:49.245196106 (508h24m49.116279151s) [noise=true , address=77.90.31.16:4735 , piece_id=XRANRGJZKUQQNWBDN44BSCHFQCDL3L6CZJXCSTXWSKF37WLQY4LA, seg_pos={0 0}, upload=446139]: Long tail cancellation due to optimal threshold Dec 26 19:53:49.245320854 (508h24m49.116403889s) [seg_pos={0 0}, upload=446139]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 19:53:49.245394545 (508h24m49.11647759s) [upload=446139]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:53:49.245400776 (508h24m49.116483811s) [upload=446139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:53:49.575129807 (508h24m49.446212842s) [upload=446139]: Done uploading segments... Dec 26 19:53:49.575150035 (508h24m49.44623307s) [upload=446139]: Upload finished. err: <nil> Duration: 30.00577677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.858784166s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 19:54:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 15.858826316s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)