Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 351949 reports (Page 197 of 7039)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [XXXX..X......X........X...X..XX..X.X......XXX.X.X..X........]
duration: 
5.942647551s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 03:00:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 5.942680824s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
6 out of 60 is failed (threshold=6) [.....X...................................X...X.X.XX.........]
duration: 
3.459907163s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:59:44 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.459961487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X.......]
duration: 
5.951774948s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:59:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 5.951802109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..............X.............X................X......X.......]
duration: 
27.405894104s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:59:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 27.406073867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
6 out of 60 is failed (threshold=6) [......X...................................X...X.X.XX........]
duration: 
3.407334576s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:59:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.40736286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...............X.............X................X......X......]
duration: 
8.30799854s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:58:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 8.30801923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X......]
duration: 
8.371669683s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:58:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 8.371695403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [X......X...................................X...X.X.XX.......]
duration: 
4.551141172s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:58:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.55119786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X.....]
duration: 
5.976911241s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:57:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 5.976968259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [................X.............X................X......X.....]
duration: 
15.115560273s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:57:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 15.115605427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.X......X...................................X...X.X.XX......]
duration: 
3.492546922s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:57:34 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.492596345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.................X.............X................X......X....]
duration: 
42.962466446s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:57:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 42.962487064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X....]
duration: 
6.015889132s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:56:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 6.015921604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..................X.............X................X......X...]
duration: 
36.71008998s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:56:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 36.710115158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [..X......X...................................X...X.X.XX.....]
duration: 
3.324184052s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:56:28 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.324211805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X...]
duration: 
5.710686746s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:56:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 5.710712425s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [...X......X...................................X...X.X.XX....]
duration: 
4.245944933s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:55:51 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.245970231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...................X.............X................X......X..]
duration: 
6.820872399s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:55:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 6.820894662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [XX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X..]
duration: 
5.888856935s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:55:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 5.888884076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X.]
duration: 
6.003559196s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:55:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 6.003577531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [....................X.............X................X......X.]
duration: 
31.53097364s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:55:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 31.531003997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [....X......X...................................X...X.X.XX...]
duration: 
4.418743507s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:55:01 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.418778704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..X]
duration: 
30.003351699s
trace: 
https://jaeger.dev.storj.io/trace/42355d6fffeb67eb
FAIL (2025-12-30 02:54:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ) context deadline exceeded
======LOGS======
Dec 30 02:54:03.695732161 (587h25m3.566815207s) [upload=514944]: Starting upload
Dec 30 02:54:03.695839806 (587h25m3.566922851s) [upload=514944]: Done starting upload
Dec 30 02:54:03.695951380 (587h25m3.567034435s) [upload=514944]: Uploading segments...
Dec 30 02:54:03.696153684 (587h25m3.567236729s) [upload=514944]: Got next segment. Inline: false
Dec 30 02:54:03.696208709 (587h25m3.567291754s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:54:03.696215853 (587h25m3.567298898s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:54:03.940794520 (587h25m3.811877565s) [seg_pos={0 0}, upload=514944]: Begin upload segment...
Dec 30 02:54:03.940857278 (587h25m3.811940323s) [seg_pos={0 0}, upload=514944]: Done begin upload segment.
Dec 30 02:54:03.940895731 (587h25m3.811978767s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.940965816 (587h25m3.812048851s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942358206 (587h25m3.813441251s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.942884248 (587h25m3.813967293s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943339756 (587h25m3.814422801s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.943746079 (587h25m3.814829124s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944223328 (587h25m3.815306373s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.944693373 (587h25m3.815776408s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946441131 (587h25m3.817524176s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.946632205 (587h25m3.81771525s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.950286153 (587h25m3.821369228s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.952237358 (587h25m3.823320403s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.954128789 (587h25m3.825211834s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.955783911 (587h25m3.826866957s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958216433 (587h25m3.829299509s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:03.958747645 (587h25m3.82983068s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Uploading piece...
Dec 30 02:54:05.542884847 (587h25m5.413967892s) [upload=514944]: Next returned nil segment
Dec 30 02:54:05.542900306 (587h25m5.413983351s) [upload=514944]: Waiting for error group managing segments...
Dec 30 02:54:05.923434604 (587h25m5.794517649s) [noise=true , address=145.79.227.11:4729   , piece_id=TF6MEG6J2RACVF3NK57AYYGPSOXKIRFOF4EVXNDAGKOJK4MYMSSQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.923809197 (587h25m5.794892242s) [noise=true , address=145.79.227.28:4727   , piece_id=O6UBDCPDGLPYOB4HNCK4FBRJHRSMFCHOUGBVSM36RPSVSTFLAL4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.937024401 (587h25m5.808107446s) [noise=true , address=145.79.227.46:4601   , piece_id=5IUX4R47XOJPS3RIHYMPXX2JPRK2VOML7CNUEXNKKBUQGTVVZNCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.941152270 (587h25m5.812235315s) [noise=true , address=145.79.227.38:4979   , piece_id=QQZRJGB2T6K7W6M5QIPB5AVUOBVUXCSOQ63FSJBDEFHS2ORXJOCA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.942508673 (587h25m5.813591718s) [noise=true , address=145.79.227.45:4231   , piece_id=G3Y3FQHP2EUOTATL3ZRSUYMI6GTNCWFOEET557N5Z7GIH7BY5SEA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944587011 (587h25m5.815670056s) [noise=true , address=145.79.227.3:4150    , piece_id=RFZ46GIBFEFHWXYICLPLOJF4VAOFXVTCZHBPS2B4TRAPXQUY4S6A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.944606198 (587h25m5.815689233s) [noise=true , address=145.79.227.15:4876   , piece_id=IXYLMEPNDG5IGC3UEIWMCB4PT7UUEP4HJVP6GIDKPDGO7F6BD6AA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.951836786 (587h25m5.822919831s) [noise=true , address=145.79.227.17:4971   , piece_id=JDWQWGKJG3GUFGPWTES2EOY7VROMI6AI45T7FWIOBMFZVQWBKLZQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.955872009 (587h25m5.826955055s) [noise=true , address=145.79.227.6:4259    , piece_id=IJCT46JR7IISIBBILNJISTYARWPPP6DRXJWQXIKO2HCBIEKOAEXA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.956697782 (587h25m5.827780827s) [noise=true , address=145.79.227.2:4113    , piece_id=UDIQOKUFPZHFBX2RKABTZ3HQATE37SLTIQ5BN2TJYXUAT4466ODQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:05.983783998 (587h25m5.854867043s) [noise=true , address=145.79.227.22:4509   , piece_id=JTTJCXUJSOFTSTTL5OAZEPULA6OA3KZ4STKRCJB6IND2S5IQR3MA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.209962347 (587h25m6.081045392s) [noise=true , address=77.90.31.16:4736     , piece_id=EBKLWOOMSFCEPJVSFTOHMMOKHKYEOIFTCXFBULGTC5XGOU7YAG4A, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222423354 (587h25m6.093506399s) [noise=true , address=45.91.116.6:4973     , piece_id=ROOAFYYUG5A6BTTK342FRMOVYIH2MJW7H2PWJURVK44ROCNEOOFQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: <nil>
Dec 30 02:54:06.222450887 (587h25m6.093533932s) [seg_pos={0 0}, upload=514944]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:54:06.222748975 (587h25m6.09383203s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222758453 (587h25m6.093841488s) [noise=true , address=45.91.116.4:4937     , piece_id=CALC343NWUNCYNGLBBB5M4PWSR26GTWKJFF7A3JF5UHEAURNC2LQ, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.222888480 (587h25m6.093971525s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 02:54:06.222924118 (587h25m6.094007153s) [noise=true , address=77.90.31.21:4878     , piece_id=P3BWZ5HIPTCE5ULMUB67PZIAPK2VC6KLP4ZPC5J3YCBIRYBDXLNA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223114150 (587h25m6.094197185s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36774->45.91.116.9:4264: use of closed network connection
Dec 30 02:54:06.223124360 (587h25m6.094207395s) [noise=true , address=45.91.116.9:4264     , piece_id=TFNANGRF2C4GA6XWEP6XQSGLJ4JGZSOHK3QEBEGSC62RYPHFVMIA, seg_pos={0 0}, upload=514944]: Long tail cancellation due to optimal threshold
Dec 30 02:54:06.223148615 (587h25m6.09423165s) [seg_pos={0 0}, upload=514944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:54:06.223263905 (587h25m6.094346951s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:54:06.223272732 (587h25m6.094355767s) [upload=514944]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:54:06.527482886 (587h25m6.398565921s) [upload=514944]: Done uploading segments...
Dec 30 02:54:06.527511160 (587h25m6.398594195s) [upload=514944]: Upload finished. err: <nil>
Duration: 30.003387888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.....................X.............X................X......X]
duration: 
59.271401872s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 02:54:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 59.271414066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.....X......X...................................X...X.X.XX..]
duration: 
4.007213508s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:54:00 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.007357712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X..]
duration: 
5.872202388s
trace: 
https://jaeger.dev.storj.io/trace/1cb8a77306ab6d3e
FAIL (2025-12-30 02:53:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ) context deadline exceeded
======LOGS======
Dec 30 02:51:32.972607085 (587h22m32.84369012s) [upload=514911]: Starting upload
Dec 30 02:51:32.972667029 (587h22m32.843750074s) [upload=514911]: Done starting upload
Dec 30 02:51:32.972788069 (587h22m32.843871104s) [upload=514911]: Uploading segments...
Dec 30 02:51:32.973139108 (587h22m32.844222153s) [upload=514911]: Got next segment. Inline: false
Dec 30 02:51:32.973177952 (587h22m32.844260987s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:51:32.973182571 (587h22m32.844265616s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:51:33.215678993 (587h22m33.086762038s) [seg_pos={0 0}, upload=514911]: Begin upload segment...
Dec 30 02:51:33.215729739 (587h22m33.086812774s) [seg_pos={0 0}, upload=514911]: Done begin upload segment.
Dec 30 02:51:33.215754597 (587h22m33.086837632s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.215893852 (587h22m33.086976897s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216520665 (587h22m33.08760371s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216530694 (587h22m33.087613729s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216881652 (587h22m33.087964697s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216937209 (587h22m33.088020254s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217234985 (587h22m33.08831802s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217321801 (587h22m33.088404846s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217540878 (587h22m33.088623913s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217691644 (587h22m33.08877469s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217897206 (587h22m33.088980241s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218286066 (587h22m33.089369111s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218303029 (587h22m33.089386064s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.220000421 (587h22m33.091083467s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.223346182 (587h22m33.094429217s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.224423794 (587h22m33.095506829s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:34.921684984 (587h22m34.792768019s) [upload=514911]: Next returned nil segment
Dec 30 02:51:34.921718488 (587h22m34.792801533s) [upload=514911]: Waiting for error group managing segments...
Dec 30 02:51:35.205929951 (587h22m35.077012986s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.206471341 (587h22m35.077554376s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210284042 (587h22m35.081367077s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210893311 (587h22m35.081976346s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.216280619 (587h22m35.087363654s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.223371752 (587h22m35.094454797s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.228885371 (587h22m35.099968407s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.233288746 (587h22m35.104371791s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.297598121 (587h22m35.168681156s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.498669702 (587h22m35.369752737s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.499400724 (587h22m35.370483769s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.504573445 (587h22m35.37565649s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505893978 (587h22m35.376977023s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505907044 (587h22m35.376990089s) [seg_pos={0 0}, upload=514911]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:51:35.506273220 (587h22m35.377356286s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506285234 (587h22m35.377368279s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506605393 (587h22m35.377688428s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506615102 (587h22m35.377698147s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506777561 (587h22m35.377860616s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506786738 (587h22m35.377869784s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506812929 (587h22m35.377895974s) [seg_pos={0 0}, upload=514911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:51:35.506920573 (587h22m35.378003609s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:51:35.506929530 (587h22m35.378012585s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:51:35.794982598 (587h22m35.666065633s) [upload=514911]: Done uploading segments...
Dec 30 02:51:35.795005902 (587h22m35.666088947s) [upload=514911]: Upload finished. err: <nil>
Duration: 5.872260177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......................X.............X................X......]
duration: 
34.639880481s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:53:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 34.639910779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......................X.............X................X.....]
duration: 
15.080329079s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:52:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 15.080342646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [......X......X...................................X...X.X.XX.]
duration: 
3.748532381s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:52:45 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.748556736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X.]
duration: 
5.757552007s
trace: 
https://jaeger.dev.storj.io/trace/1cb8a77306ab6d3e
FAIL (2025-12-30 02:52:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ) context deadline exceeded
======LOGS======
Dec 30 02:51:32.972607085 (587h22m32.84369012s) [upload=514911]: Starting upload
Dec 30 02:51:32.972667029 (587h22m32.843750074s) [upload=514911]: Done starting upload
Dec 30 02:51:32.972788069 (587h22m32.843871104s) [upload=514911]: Uploading segments...
Dec 30 02:51:32.973139108 (587h22m32.844222153s) [upload=514911]: Got next segment. Inline: false
Dec 30 02:51:32.973177952 (587h22m32.844260987s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:51:32.973182571 (587h22m32.844265616s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:51:33.215678993 (587h22m33.086762038s) [seg_pos={0 0}, upload=514911]: Begin upload segment...
Dec 30 02:51:33.215729739 (587h22m33.086812774s) [seg_pos={0 0}, upload=514911]: Done begin upload segment.
Dec 30 02:51:33.215754597 (587h22m33.086837632s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.215893852 (587h22m33.086976897s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216520665 (587h22m33.08760371s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216530694 (587h22m33.087613729s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216881652 (587h22m33.087964697s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216937209 (587h22m33.088020254s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217234985 (587h22m33.08831802s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217321801 (587h22m33.088404846s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217540878 (587h22m33.088623913s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217691644 (587h22m33.08877469s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217897206 (587h22m33.088980241s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218286066 (587h22m33.089369111s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218303029 (587h22m33.089386064s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.220000421 (587h22m33.091083467s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.223346182 (587h22m33.094429217s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.224423794 (587h22m33.095506829s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:34.921684984 (587h22m34.792768019s) [upload=514911]: Next returned nil segment
Dec 30 02:51:34.921718488 (587h22m34.792801533s) [upload=514911]: Waiting for error group managing segments...
Dec 30 02:51:35.205929951 (587h22m35.077012986s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.206471341 (587h22m35.077554376s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210284042 (587h22m35.081367077s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210893311 (587h22m35.081976346s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.216280619 (587h22m35.087363654s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.223371752 (587h22m35.094454797s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.228885371 (587h22m35.099968407s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.233288746 (587h22m35.104371791s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.297598121 (587h22m35.168681156s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.498669702 (587h22m35.369752737s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.499400724 (587h22m35.370483769s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.504573445 (587h22m35.37565649s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505893978 (587h22m35.376977023s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505907044 (587h22m35.376990089s) [seg_pos={0 0}, upload=514911]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:51:35.506273220 (587h22m35.377356286s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506285234 (587h22m35.377368279s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506605393 (587h22m35.377688428s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506615102 (587h22m35.377698147s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506777561 (587h22m35.377860616s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506786738 (587h22m35.377869784s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506812929 (587h22m35.377895974s) [seg_pos={0 0}, upload=514911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:51:35.506920573 (587h22m35.378003609s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:51:35.506929530 (587h22m35.378012585s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:51:35.794982598 (587h22m35.666065633s) [upload=514911]: Done uploading segments...
Dec 30 02:51:35.795005902 (587h22m35.666088947s) [upload=514911]: Upload finished. err: <nil>
Duration: 5.757581765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [........................X.............X................X....]
duration: 
44.924166105s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:52:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 44.924195911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.......X......X...................................X...X.X.XX]
duration: 
5.001374027s
trace: 
https://jaeger.dev.storj.io/trace/14b283cf2d9b0313
FAIL (2025-12-30 02:52:17 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001397421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.X]
duration: 
30.003385068s
trace: 
https://jaeger.dev.storj.io/trace/1cb8a77306ab6d3e
FAIL (2025-12-30 02:52:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ) context deadline exceeded
======LOGS======
Dec 30 02:51:32.972607085 (587h22m32.84369012s) [upload=514911]: Starting upload
Dec 30 02:51:32.972667029 (587h22m32.843750074s) [upload=514911]: Done starting upload
Dec 30 02:51:32.972788069 (587h22m32.843871104s) [upload=514911]: Uploading segments...
Dec 30 02:51:32.973139108 (587h22m32.844222153s) [upload=514911]: Got next segment. Inline: false
Dec 30 02:51:32.973177952 (587h22m32.844260987s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:51:32.973182571 (587h22m32.844265616s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:51:33.215678993 (587h22m33.086762038s) [seg_pos={0 0}, upload=514911]: Begin upload segment...
Dec 30 02:51:33.215729739 (587h22m33.086812774s) [seg_pos={0 0}, upload=514911]: Done begin upload segment.
Dec 30 02:51:33.215754597 (587h22m33.086837632s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.215893852 (587h22m33.086976897s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216520665 (587h22m33.08760371s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216530694 (587h22m33.087613729s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216881652 (587h22m33.087964697s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.216937209 (587h22m33.088020254s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217234985 (587h22m33.08831802s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217321801 (587h22m33.088404846s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217540878 (587h22m33.088623913s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217691644 (587h22m33.08877469s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.217897206 (587h22m33.088980241s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218286066 (587h22m33.089369111s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.218303029 (587h22m33.089386064s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.220000421 (587h22m33.091083467s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.223346182 (587h22m33.094429217s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:33.224423794 (587h22m33.095506829s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Uploading piece...
Dec 30 02:51:34.921684984 (587h22m34.792768019s) [upload=514911]: Next returned nil segment
Dec 30 02:51:34.921718488 (587h22m34.792801533s) [upload=514911]: Waiting for error group managing segments...
Dec 30 02:51:35.205929951 (587h22m35.077012986s) [noise=true , address=145.79.227.17:4972   , piece_id=S37D5RFSTPCDSOOHGKEXCFEAGYUTJ2WR7AL6NTHERV5XXWZPHVQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.206471341 (587h22m35.077554376s) [noise=true , address=145.79.227.36:4518   , piece_id=KAZ66ZAR24SKOKU56YDSNF7MRCVZ4VFWZNSYWZI3TLRRKZUT4KOA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210284042 (587h22m35.081367077s) [noise=true , address=145.79.227.4:4186    , piece_id=IB7VMRDEFIFKVNOVPW7YY23W2KXC5QBPHEVICNL657FWWPFPU4BA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.210893311 (587h22m35.081976346s) [noise=true , address=145.79.227.16:4911   , piece_id=QFBQLHPPQZN2I25O4S6ZMVMFOJ5PFITZXO4OCUBRJ3OQQYNQLBJQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.216280619 (587h22m35.087363654s) [noise=true , address=145.79.227.15:4876   , piece_id=HMWMDPU7OYR3YM6JFID4DTL5QABGGKF53VSUDQKZFZPEVSOYEN3A, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.223371752 (587h22m35.094454797s) [noise=true , address=145.79.227.7:4293    , piece_id=3M3VU3XFWYYMZSXSKJ3RTYP77CTLDPKOL62Y6X4JAKVBMH2G2IQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.228885371 (587h22m35.099968407s) [noise=true , address=145.79.227.42:4121   , piece_id=M2SOXBOG3BQA2D727XGJ63HRTCT7BUD6XTWYOGUDMUGFOUK6LIQQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.233288746 (587h22m35.104371791s) [noise=true , address=145.79.227.5:4221    , piece_id=IBKGOAL4EZSWFGEJL5WLAF4XHF2OISPUI4WBGKKJCEKTPLHEM2TA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.297598121 (587h22m35.168681156s) [noise=true , address=145.79.227.38:4977   , piece_id=BGOOVLNI4PCCUNTN4UXMUHTMGYFPRS74LISUVPC46ABLBIW6JPNQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.498669702 (587h22m35.369752737s) [noise=true , address=45.91.116.7:4009     , piece_id=3DF7HMG2GX6IMMORKQP7XUJM3L63MAT6BZ2BB2UEP7NDMIT5E5QQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.499400724 (587h22m35.370483769s) [noise=true , address=77.90.31.15:4697     , piece_id=75I22CPQ56KBRGO2ACHDNGVS2NYHZ3QMQ2EWTGJQKARDJVV42FKA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.504573445 (587h22m35.37565649s) [noise=true , address=77.90.31.11:4555     , piece_id=BGH3JKDPLAPXYHGCPYSUKFD5HVV2BCEBPCRGCYMQYO4VRIZZYSDQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505893978 (587h22m35.376977023s) [noise=true , address=45.91.116.9:4263     , piece_id=FNKU74A2DCTJU4CLHRZ5OETDNAPSCD5YBQUNV3NHISQZFJP5RG7Q, seg_pos={0 0}, upload=514911]: Done uploading piece. err: <nil>
Dec 30 02:51:35.505907044 (587h22m35.376990089s) [seg_pos={0 0}, upload=514911]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:51:35.506273220 (587h22m35.377356286s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506285234 (587h22m35.377368279s) [noise=true , address=45.91.116.13:4156    , piece_id=3HI545G5GZRR6NAFGR2GGHTW6G2BBZWNDYWX32MX5744QOCLR7MA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506605393 (587h22m35.377688428s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506615102 (587h22m35.377698147s) [noise=true , address=77.90.31.12:4590     , piece_id=MRS2NSJ7ELQVIWXFIJC7Y2N2DXMTVMGSVYYCOSSHZLBTQ7NYB6OA, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506777561 (587h22m35.377860616s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:51:35.506786738 (587h22m35.377869784s) [noise=true , address=45.91.116.12:4227    , piece_id=F3PMSKB5ADKGKRNRFPCCUD7UJC6O4O6WCYCNJUVBXE6SXUFH5OMQ, seg_pos={0 0}, upload=514911]: Long tail cancellation due to optimal threshold
Dec 30 02:51:35.506812929 (587h22m35.377895974s) [seg_pos={0 0}, upload=514911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:51:35.506920573 (587h22m35.378003609s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:51:35.506929530 (587h22m35.378012585s) [upload=514911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:51:35.794982598 (587h22m35.666065633s) [upload=514911]: Done uploading segments...
Dec 30 02:51:35.795005902 (587h22m35.666088947s) [upload=514911]: Upload finished. err: <nil>
Duration: 30.003410516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.........................X.............X................X...]
duration: 
33.583333581s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:51:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 33.583372285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
6 out of 60 is failed (threshold=6) [........X......X...................................X...X.X.X]
duration: 
5.001963583s
trace: 
https://jaeger.dev.storj.io/trace/1edbdb3c006e63f4
FAIL (2025-12-30 02:51:09 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001987338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X.]
duration: 
5.82056162s
trace: 
https://jaeger.dev.storj.io/trace/262c3bd6c57bf2fd
FAIL (2025-12-30 02:51:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA) context deadline exceeded
======LOGS======
Dec 30 02:50:20.963865240 (587h21m20.834948285s) [upload=514892]: Starting upload
Dec 30 02:50:20.964006930 (587h21m20.835090005s) [upload=514892]: Done starting upload
Dec 30 02:50:20.964269480 (587h21m20.835352515s) [upload=514892]: Uploading segments...
Dec 30 02:50:20.964332240 (587h21m20.835415285s) [upload=514892]: Got next segment. Inline: false
Dec 30 02:50:20.964434605 (587h21m20.83551766s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:50:20.964444023 (587h21m20.835527068s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:50:21.213344468 (587h21m21.084427523s) [seg_pos={0 0}, upload=514892]: Begin upload segment...
Dec 30 02:50:21.213405384 (587h21m21.084488439s) [seg_pos={0 0}, upload=514892]: Done begin upload segment.
Dec 30 02:50:21.213435722 (587h21m21.084518767s) [noise=true , address=145.79.227.15:4876   , piece_id=RNJGK3J2VJJACUVZD2CHOJZNCFGDBPAPB3YVYO3XKXUUZN3X3QBA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.213501768 (587h21m21.084584813s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.214033019 (587h21m21.085116054s) [noise=true , address=145.79.227.33:4870   , piece_id=7BUCXCDLWT2AKBISFXX6Z2OYRUYVGTALQDOPOGQD7PKEIDBIQ2YA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.214779300 (587h21m21.085862345s) [noise=true , address=145.79.227.17:4970   , piece_id=APWY64VM25WQQVA3FDS32YSTPOGNYQNKXEQYJKZLMD7BX2O53IWA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.215409910 (587h21m21.086492956s) [noise=true , address=145.79.227.48:4301   , piece_id=P6DGWJVLB6SJ65SU2I4Y6FG2BTXXYZWTWFRZPHYWDGJBW5WUTRUQ, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.216042756 (587h21m21.087125801s) [noise=true , address=45.91.116.13:4153    , piece_id=JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.216624984 (587h21m21.087708019s) [noise=true , address=77.90.31.21:4877     , piece_id=LMDGUUJKE774LJBYKCO2I35PMO2EJHSGSBQCBD2UOIWO3QENC57Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217194389 (587h21m21.088277434s) [noise=true , address=145.79.227.7:4295    , piece_id=6BEAXFLX2V5CMWHJZPOVB6LOMSZMYIUEI24S4BRTOVQCVTW6HDLA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217871278 (587h21m21.088954323s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217946441 (587h21m21.089029486s) [noise=true , address=145.79.227.43:4159   , piece_id=Z7SBPARFMAQHG24ZHKY3SFCBRXM3WJZHJLX2XIDYCSMEDBEHZGJA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218480378 (587h21m21.089563423s) [noise=true , address=145.79.227.52:4445   , piece_id=UJHWTCMQQZ7AQGQDIMU6TFKEYHI6U3HJQMG2JKQXLP6UENT3ZF7A, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218501568 (587h21m21.089584613s) [noise=true , address=145.79.227.4:4186    , piece_id=F6SA3M3YFP2CUMFYNU76YQDCS7VSATIOV26KFGOIXEGMDOY4UFTA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218862115 (587h21m21.08994516s) [noise=true , address=45.91.116.5:4297     , piece_id=AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218968187 (587h21m21.090051232s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.219189498 (587h21m21.090272533s) [noise=true , address=77.90.31.10:4520     , piece_id=JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.219529425 (587h21m21.09061247s) [noise=true , address=77.90.31.14:4663     , piece_id=SK43LBAPRRLRGIEBVNXVOYXHSCO6QNXJUK3O6HS7LFE7KBJVCAFA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:22.906863774 (587h21m22.777946819s) [upload=514892]: Next returned nil segment
Dec 30 02:50:22.906880355 (587h21m22.7779634s) [upload=514892]: Waiting for error group managing segments...
Dec 30 02:50:23.190665917 (587h21m23.061748952s) [noise=true , address=145.79.227.48:4301   , piece_id=P6DGWJVLB6SJ65SU2I4Y6FG2BTXXYZWTWFRZPHYWDGJBW5WUTRUQ, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.198999546 (587h21m23.070082591s) [noise=true , address=145.79.227.15:4876   , piece_id=RNJGK3J2VJJACUVZD2CHOJZNCFGDBPAPB3YVYO3XKXUUZN3X3QBA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.203043756 (587h21m23.074126801s) [noise=true , address=145.79.227.17:4970   , piece_id=APWY64VM25WQQVA3FDS32YSTPOGNYQNKXEQYJKZLMD7BX2O53IWA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.203570609 (587h21m23.074653644s) [noise=true , address=145.79.227.7:4295    , piece_id=6BEAXFLX2V5CMWHJZPOVB6LOMSZMYIUEI24S4BRTOVQCVTW6HDLA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.204073528 (587h21m23.075156573s) [noise=true , address=145.79.227.4:4186    , piece_id=F6SA3M3YFP2CUMFYNU76YQDCS7VSATIOV26KFGOIXEGMDOY4UFTA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.209255645 (587h21m23.08033869s) [noise=true , address=145.79.227.52:4445   , piece_id=UJHWTCMQQZ7AQGQDIMU6TFKEYHI6U3HJQMG2JKQXLP6UENT3ZF7A, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.211694950 (587h21m23.082777985s) [noise=true , address=145.79.227.33:4870   , piece_id=7BUCXCDLWT2AKBISFXX6Z2OYRUYVGTALQDOPOGQD7PKEIDBIQ2YA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.213285338 (587h21m23.084368393s) [noise=true , address=145.79.227.43:4159   , piece_id=Z7SBPARFMAQHG24ZHKY3SFCBRXM3WJZHJLX2XIDYCSMEDBEHZGJA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.392374462 (587h21m23.263457507s) [noise=true , address=77.90.31.21:4877     , piece_id=LMDGUUJKE774LJBYKCO2I35PMO2EJHSGSBQCBD2UOIWO3QENC57Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.399544064 (587h21m23.270627109s) [noise=true , address=77.90.31.10:4520     , piece_id=JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.412630062 (587h21m23.283713107s) [noise=true , address=77.90.31.14:4663     , piece_id=SK43LBAPRRLRGIEBVNXVOYXHSCO6QNXJUK3O6HS7LFE7KBJVCAFA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.545158888 (587h21m23.416241923s) [noise=true , address=45.91.116.5:4297     , piece_id=AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:24.246508326 (587h21m24.117591371s) [noise=true , address=45.91.116.13:4153    , piece_id=JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:24.246530818 (587h21m24.117613863s) [seg_pos={0 0}, upload=514892]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:50:24.246764743 (587h21m24.117847779s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.246774512 (587h21m24.117857547s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.246932183 (587h21m24.118015228s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.246954494 (587h21m24.118037559s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.246999371 (587h21m24.118082416s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.247012034 (587h21m24.118095069s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.247033726 (587h21m24.118116771s) [seg_pos={0 0}, upload=514892]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:50:24.247150949 (587h21m24.118233994s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:50:24.247159365 (587h21m24.11824241s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:50:24.531328337 (587h21m24.402411392s) [upload=514892]: Done uploading segments...
Dec 30 02:50:24.531356491 (587h21m24.402439536s) [upload=514892]: Upload finished. err: <nil>
Duration: 5.820582981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.X]
duration: 
30.002210016s
trace: 
https://jaeger.dev.storj.io/trace/262c3bd6c57bf2fd
FAIL (2025-12-30 02:50:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA) context deadline exceeded
======LOGS======
Dec 30 02:50:20.963865240 (587h21m20.834948285s) [upload=514892]: Starting upload
Dec 30 02:50:20.964006930 (587h21m20.835090005s) [upload=514892]: Done starting upload
Dec 30 02:50:20.964269480 (587h21m20.835352515s) [upload=514892]: Uploading segments...
Dec 30 02:50:20.964332240 (587h21m20.835415285s) [upload=514892]: Got next segment. Inline: false
Dec 30 02:50:20.964434605 (587h21m20.83551766s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:50:20.964444023 (587h21m20.835527068s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:50:21.213344468 (587h21m21.084427523s) [seg_pos={0 0}, upload=514892]: Begin upload segment...
Dec 30 02:50:21.213405384 (587h21m21.084488439s) [seg_pos={0 0}, upload=514892]: Done begin upload segment.
Dec 30 02:50:21.213435722 (587h21m21.084518767s) [noise=true , address=145.79.227.15:4876   , piece_id=RNJGK3J2VJJACUVZD2CHOJZNCFGDBPAPB3YVYO3XKXUUZN3X3QBA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.213501768 (587h21m21.084584813s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.214033019 (587h21m21.085116054s) [noise=true , address=145.79.227.33:4870   , piece_id=7BUCXCDLWT2AKBISFXX6Z2OYRUYVGTALQDOPOGQD7PKEIDBIQ2YA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.214779300 (587h21m21.085862345s) [noise=true , address=145.79.227.17:4970   , piece_id=APWY64VM25WQQVA3FDS32YSTPOGNYQNKXEQYJKZLMD7BX2O53IWA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.215409910 (587h21m21.086492956s) [noise=true , address=145.79.227.48:4301   , piece_id=P6DGWJVLB6SJ65SU2I4Y6FG2BTXXYZWTWFRZPHYWDGJBW5WUTRUQ, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.216042756 (587h21m21.087125801s) [noise=true , address=45.91.116.13:4153    , piece_id=JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.216624984 (587h21m21.087708019s) [noise=true , address=77.90.31.21:4877     , piece_id=LMDGUUJKE774LJBYKCO2I35PMO2EJHSGSBQCBD2UOIWO3QENC57Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217194389 (587h21m21.088277434s) [noise=true , address=145.79.227.7:4295    , piece_id=6BEAXFLX2V5CMWHJZPOVB6LOMSZMYIUEI24S4BRTOVQCVTW6HDLA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217871278 (587h21m21.088954323s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.217946441 (587h21m21.089029486s) [noise=true , address=145.79.227.43:4159   , piece_id=Z7SBPARFMAQHG24ZHKY3SFCBRXM3WJZHJLX2XIDYCSMEDBEHZGJA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218480378 (587h21m21.089563423s) [noise=true , address=145.79.227.52:4445   , piece_id=UJHWTCMQQZ7AQGQDIMU6TFKEYHI6U3HJQMG2JKQXLP6UENT3ZF7A, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218501568 (587h21m21.089584613s) [noise=true , address=145.79.227.4:4186    , piece_id=F6SA3M3YFP2CUMFYNU76YQDCS7VSATIOV26KFGOIXEGMDOY4UFTA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218862115 (587h21m21.08994516s) [noise=true , address=45.91.116.5:4297     , piece_id=AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.218968187 (587h21m21.090051232s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.219189498 (587h21m21.090272533s) [noise=true , address=77.90.31.10:4520     , piece_id=JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:21.219529425 (587h21m21.09061247s) [noise=true , address=77.90.31.14:4663     , piece_id=SK43LBAPRRLRGIEBVNXVOYXHSCO6QNXJUK3O6HS7LFE7KBJVCAFA, seg_pos={0 0}, upload=514892]: Uploading piece...
Dec 30 02:50:22.906863774 (587h21m22.777946819s) [upload=514892]: Next returned nil segment
Dec 30 02:50:22.906880355 (587h21m22.7779634s) [upload=514892]: Waiting for error group managing segments...
Dec 30 02:50:23.190665917 (587h21m23.061748952s) [noise=true , address=145.79.227.48:4301   , piece_id=P6DGWJVLB6SJ65SU2I4Y6FG2BTXXYZWTWFRZPHYWDGJBW5WUTRUQ, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.198999546 (587h21m23.070082591s) [noise=true , address=145.79.227.15:4876   , piece_id=RNJGK3J2VJJACUVZD2CHOJZNCFGDBPAPB3YVYO3XKXUUZN3X3QBA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.203043756 (587h21m23.074126801s) [noise=true , address=145.79.227.17:4970   , piece_id=APWY64VM25WQQVA3FDS32YSTPOGNYQNKXEQYJKZLMD7BX2O53IWA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.203570609 (587h21m23.074653644s) [noise=true , address=145.79.227.7:4295    , piece_id=6BEAXFLX2V5CMWHJZPOVB6LOMSZMYIUEI24S4BRTOVQCVTW6HDLA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.204073528 (587h21m23.075156573s) [noise=true , address=145.79.227.4:4186    , piece_id=F6SA3M3YFP2CUMFYNU76YQDCS7VSATIOV26KFGOIXEGMDOY4UFTA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.209255645 (587h21m23.08033869s) [noise=true , address=145.79.227.52:4445   , piece_id=UJHWTCMQQZ7AQGQDIMU6TFKEYHI6U3HJQMG2JKQXLP6UENT3ZF7A, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.211694950 (587h21m23.082777985s) [noise=true , address=145.79.227.33:4870   , piece_id=7BUCXCDLWT2AKBISFXX6Z2OYRUYVGTALQDOPOGQD7PKEIDBIQ2YA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.213285338 (587h21m23.084368393s) [noise=true , address=145.79.227.43:4159   , piece_id=Z7SBPARFMAQHG24ZHKY3SFCBRXM3WJZHJLX2XIDYCSMEDBEHZGJA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.392374462 (587h21m23.263457507s) [noise=true , address=77.90.31.21:4877     , piece_id=LMDGUUJKE774LJBYKCO2I35PMO2EJHSGSBQCBD2UOIWO3QENC57Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.399544064 (587h21m23.270627109s) [noise=true , address=77.90.31.10:4520     , piece_id=JGHLJI2HIJLUVDXMMRA6BQWPC6BESPD2SJP2QMJGNZMEPOJ5FZXA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.412630062 (587h21m23.283713107s) [noise=true , address=77.90.31.14:4663     , piece_id=SK43LBAPRRLRGIEBVNXVOYXHSCO6QNXJUK3O6HS7LFE7KBJVCAFA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:23.545158888 (587h21m23.416241923s) [noise=true , address=45.91.116.5:4297     , piece_id=AF3WFJVXSDTQB5GQIALYFAPBRIYMAA4U75RFIX7J737JDLJ72WYA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:24.246508326 (587h21m24.117591371s) [noise=true , address=45.91.116.13:4153    , piece_id=JKAC33O744CZC7WJC7K6AARQT65HUZA7LBNMFQQRJRTURHK7YO7A, seg_pos={0 0}, upload=514892]: Done uploading piece. err: <nil>
Dec 30 02:50:24.246530818 (587h21m24.117613863s) [seg_pos={0 0}, upload=514892]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:50:24.246764743 (587h21m24.117847779s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.246774512 (587h21m24.117857547s) [noise=true , address=145.79.227.35:4483   , piece_id=JBH2ME5MLGHYPCHEH4GZAQP6XMX7EVQIXM5TVETQBFMUKDCSI7HA, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.246932183 (587h21m24.118015228s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.246954494 (587h21m24.118037559s) [noise=true , address=145.79.227.34:4906   , piece_id=CTKQGQDJTYEJPH5XPUGSK4BJ4TNHVM6JJ3QTCKRSSP2BXKNVQP7Q, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.246999371 (587h21m24.118082416s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:50:24.247012034 (587h21m24.118095069s) [noise=true , address=77.90.31.15:4697     , piece_id=KPNV3ADA5C73COYEW4J4U3M4VIMYNQ4UXHZXLEWMH3L24D2JYG6Q, seg_pos={0 0}, upload=514892]: Long tail cancellation due to optimal threshold
Dec 30 02:50:24.247033726 (587h21m24.118116771s) [seg_pos={0 0}, upload=514892]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:50:24.247150949 (587h21m24.118233994s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:50:24.247159365 (587h21m24.11824241s) [upload=514892]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:50:24.531328337 (587h21m24.402411392s) [upload=514892]: Done uploading segments...
Dec 30 02:50:24.531356491 (587h21m24.402439536s) [upload=514892]: Upload finished. err: <nil>
Duration: 30.00223332s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
5 out of 60 is failed (threshold=6) [.........X......X...................................X...X.X.]
duration: 
3.472211725s
trace: 
https://jaeger.dev.storj.io/trace/43a1195231742152
FAIL (2025-12-30 02:50:32 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.472236741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..........................X.............X................X..]
duration: 
34.626299761s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:50:20 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 34.626327324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...........................X.............X................X.]
duration: 
36.238884569s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:49:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 36.23890038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
5 out of 60 is failed (threshold=6) [..........X......X...................................X...X.X]
duration: 
5.001380998s
trace: 
https://jaeger.dev.storj.io/trace/43a1195231742152
FAIL (2025-12-30 02:49:37 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001404693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX.]
duration: 
5.682607534s
trace: 
https://jaeger.dev.storj.io/trace/6f3cd5d32d1bb8d1
FAIL (2025-12-30 02:49:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ) context deadline exceeded
======LOGS======
 30 02:48:34.132668071 (587h19m34.003751117s) [upload=514869]: Starting upload
Dec 30 02:48:34.132732373 (587h19m34.003815408s) [upload=514869]: Done starting upload
Dec 30 02:48:34.132856309 (587h19m34.003939355s) [upload=514869]: Uploading segments...
Dec 30 02:48:34.132958214 (587h19m34.004041259s) [upload=514869]: Got next segment. Inline: false
Dec 30 02:48:34.133005884 (587h19m34.004088929s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:48:34.133012126 (587h19m34.004095171s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:48:34.376429361 (587h19m34.247512396s) [seg_pos={0 0}, upload=514869]: Begin upload segment...
Dec 30 02:48:34.376473975 (587h19m34.24755701s) [seg_pos={0 0}, upload=514869]: Done begin upload segment.
Dec 30 02:48:34.376497621 (587h19m34.247580656s) [noise=true , address=45.91.116.6:4975     , piece_id=OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.376550021 (587h19m34.247633056s) [noise=true , address=145.79.227.17:4971   , piece_id=NHFJV7CX4WCYPCI4KQGN7IWS7IVCY4CZZ5XV73FQJCRODYWYZTCQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377016358 (587h19m34.248099404s) [noise=true , address=77.90.31.15:4697     , piece_id=2US5STRDRCBRAXTOXOSYOQNNKW2CRG7YXMJ7V3C2OLJC2MFGBKBQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377609598 (587h19m34.248692633s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377963191 (587h19m34.249046226s) [noise=true , address=145.79.227.8:4329    , piece_id=7AQ6XYA626UTWA4I5OSQSPN676AQC24S7CU7DEXGFOPAJNHW4BJA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378353735 (587h19m34.24943677s) [noise=true , address=145.79.227.18:4006   , piece_id=TOK56L3T7IWS7MNMM3IPVGPBRCCVU4IRWG47OIUB3RSLNYSXJ2SQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378638126 (587h19m34.249721161s) [noise=true , address=145.79.227.38:4977   , piece_id=SWYUWZUZ3RKHAXDLPFQFHPTCDBPNRQPQVUJOSBARAZUZHM5LUYOQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378920343 (587h19m34.250003378s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.379235735 (587h19m34.25031887s) [noise=true , address=145.79.227.28:4727   , piece_id=DELXWBPZUAK5ITQVN4TQTMW237YSYQFCIWFB7WULUOVSGFGTJU7A, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.380116522 (587h19m34.251199557s) [noise=true , address=45.91.116.8:4046     , piece_id=JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.381401599 (587h19m34.252484644s) [noise=true , address=145.79.227.7:4294    , piece_id=2Z6XZNHCCOXORXPQWBBMVKVMA75HA7AFWW6534FXBKO43MXFYX5Q, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.382511973 (587h19m34.253595008s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.383619060 (587h19m34.254702096s) [noise=true , address=145.79.227.42:4122   , piece_id=HG6O4FI2A7QJG7HU33KVWN4HFV5Z7HBGSQGYIKNKWZKTIOGQUIPQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.384847931 (587h19m34.255930966s) [noise=true , address=145.79.227.11:4729   , piece_id=PCEGG75NP63ODMSMNDQT7JRBWSCDOWDJG7YDCGJ57DUT4YD75DCA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.385989285 (587h19m34.25707232s) [noise=true , address=145.79.227.37:4565   , piece_id=7XKP6F5R5DRBGZQ5634EJZRVBLTMRBAGJ7DTKODVBPDIYVXSK4FQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.387433374 (587h19m34.258516409s) [noise=true , address=145.79.227.14:4837   , piece_id=6LWF73KBSOVQBIN6X3ZDHF5LVBKIE4NSCLVRIPCHEZDKFZMDNRPA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:36.089013448 (587h19m35.960096483s) [upload=514869]: Next returned nil segment
Dec 30 02:48:36.089043345 (587h19m35.960126411s) [upload=514869]: Waiting for error group managing segments...
Dec 30 02:48:36.364094246 (587h19m36.235177281s) [noise=true , address=145.79.227.17:4971   , piece_id=NHFJV7CX4WCYPCI4KQGN7IWS7IVCY4CZZ5XV73FQJCRODYWYZTCQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.366015063 (587h19m36.237098108s) [noise=true , address=145.79.227.14:4837   , piece_id=6LWF73KBSOVQBIN6X3ZDHF5LVBKIE4NSCLVRIPCHEZDKFZMDNRPA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.368414632 (587h19m36.239497667s) [noise=true , address=145.79.227.42:4122   , piece_id=HG6O4FI2A7QJG7HU33KVWN4HFV5Z7HBGSQGYIKNKWZKTIOGQUIPQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.378961736 (587h19m36.250044771s) [noise=true , address=145.79.227.8:4329    , piece_id=7AQ6XYA626UTWA4I5OSQSPN676AQC24S7CU7DEXGFOPAJNHW4BJA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.385773736 (587h19m36.256856771s) [noise=true , address=145.79.227.11:4729   , piece_id=PCEGG75NP63ODMSMNDQT7JRBWSCDOWDJG7YDCGJ57DUT4YD75DCA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.389077387 (587h19m36.260160422s) [noise=true , address=145.79.227.28:4727   , piece_id=DELXWBPZUAK5ITQVN4TQTMW237YSYQFCIWFB7WULUOVSGFGTJU7A, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.392734992 (587h19m36.263818027s) [noise=true , address=145.79.227.37:4565   , piece_id=7XKP6F5R5DRBGZQ5634EJZRVBLTMRBAGJ7DTKODVBPDIYVXSK4FQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.408991477 (587h19m36.280074532s) [noise=true , address=145.79.227.7:4294    , piece_id=2Z6XZNHCCOXORXPQWBBMVKVMA75HA7AFWW6534FXBKO43MXFYX5Q, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.415968873 (587h19m36.287051918s) [noise=true , address=145.79.227.18:4006   , piece_id=TOK56L3T7IWS7MNMM3IPVGPBRCCVU4IRWG47OIUB3RSLNYSXJ2SQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.521606163 (587h19m36.392689198s) [noise=true , address=145.79.227.38:4977   , piece_id=SWYUWZUZ3RKHAXDLPFQFHPTCDBPNRQPQVUJOSBARAZUZHM5LUYOQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.675426310 (587h19m36.546509355s) [noise=true , address=77.90.31.15:4697     , piece_id=2US5STRDRCBRAXTOXOSYOQNNKW2CRG7YXMJ7V3C2OLJC2MFGBKBQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.727408643 (587h19m36.598491688s) [noise=true , address=45.91.116.8:4046     , piece_id=JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.821633622 (587h19m36.692716657s) [noise=true , address=45.91.116.6:4975     , piece_id=OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.821650634 (587h19m36.692733669s) [seg_pos={0 0}, upload=514869]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:48:36.821841658 (587h19m36.692924693s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:36.821848210 (587h19m36.692931245s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.821964291 (587h19m36.693047326s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection
Dec 30 02:48:36.821969161 (587h19m36.693052196s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.822204739 (587h19m36.693287784s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:36.822240267 (587h19m36.693323302s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.822266086 (587h19m36.693349121s) [seg_pos={0 0}, upload=514869]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:48:36.822369593 (587h19m36.693452628s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:36.822374983 (587h19m36.693458018s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:37.116690878 (587h19m36.987773923s) [upload=514869]: Done uploading segments...
Dec 30 02:48:37.116712188 (587h19m36.987795223s) [upload=514869]: Upload finished. err: <nil>
Duration: 5.68262652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [............................X.............X................X]
duration: 
1m0.00329045s
trace: 
https://jaeger.dev.storj.io/trace/51b6b5b0525643dc
FAIL (2025-12-30 02:49:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
87h19m9.380950297s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.509876781 (587h19m9.380959816s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510586241 (587h19m9.381669286s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.510882736 (587h19m9.381965771s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511365575 (587h19m9.38244862s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511595954 (587h19m9.382678989s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.511749045 (587h19m9.38283209s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512187550 (587h19m9.383270585s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.512606718 (587h19m9.383689753s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513016058 (587h19m9.384099093s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513531770 (587h19m9.384614815s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.513927824 (587h19m9.385010859s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514493471 (587h19m9.385576516s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.514873925 (587h19m9.38595697s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515333681 (587h19m9.386416726s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:09.515738281 (587h19m9.386821327s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Uploading piece...
Dec 30 02:48:10.819776604 (587h19m10.690859649s) [upload=514865]: Next returned nil segment
Dec 30 02:48:10.819794729 (587h19m10.690877764s) [upload=514865]: Waiting for error group managing segments...
Dec 30 02:48:11.148202518 (587h19m11.019285553s) [noise=true , address=64.49.13.27:4962     , piece_id=POVRKMG44Z7WKRXAOYP6CXYRPDBTQKOUB4KKD6CYL5PKSI3LNSVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.166845448 (587h19m11.037928493s) [noise=true , address=64.49.13.26:4950     , piece_id=UBLUTDSJJOCA3RG4ZR64T6JJJ3TBE3RIJVHZQ522F35PQ7RUULOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.402828266 (587h19m11.273911331s) [noise=true , address=145.79.227.23:4578   , piece_id=JQWEPHSTH76ZBIFF44A2TGKF7DS24NBW6TRMUB3WXOOT633ZF4BA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.420150510 (587h19m11.291233545s) [noise=true , address=145.79.227.38:4011   , piece_id=TBBZDTBAWYKR5XBC3PU2MVS262DZPSJPN2RSFJO6OGPRP6BFNIBA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.422824031 (587h19m11.293907066s) [noise=true , address=145.79.227.18:4035   , piece_id=O4MEHNG6ZOLTHHY72B4XUR2HYXBXNFVC4IJX6OJAKXOTWNWUJZRA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.448269213 (587h19m11.319352278s) [noise=true , address=5.231.73.4:4021      , piece_id=2RFSPNQ56V6Q6SQY7RYDIQI2YOI3GEL35SW2XLPGJSCGJJASF2IQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.500668088 (587h19m11.371751123s) [noise=true , address=5.231.73.3:4019      , piece_id=IEWK2QRJIOSPIDX64P54TZDJ2YV4TOG2JC37YP75V2NBXQAFLEZQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.569424741 (587h19m11.440507786s) [noise=true , address=77.90.31.11:4574     , piece_id=MRCH3NJ2XF7FJHKWICCA3IXY752CV3JFRSTFWB55RQTIYWXOFTPQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.684475295 (587h19m11.55555834s) [noise=true , address=77.90.31.16:4767     , piece_id=L7RKL3GQ6TVVYLO23MH7PK5E6AURO24VXY4JGSNBQ66NESWB6YOA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:11.750299205 (587h19m11.62138225s) [noise=true , address=77.90.31.18:4839     , piece_id=JRCGIB5VN2PTDGOHIHXEFXSLSFT5GRX4VFD7B4ILROMVLFDFTM4A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.276826644 (587h19m12.147909689s) [noise=true , address=145.79.227.44:4222   , piece_id=VRKJ2X6G35UTA4QHXTIQU54GV4M7RM62CCMEWHAJI3LSVDTRMECQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.294938533 (587h19m12.166021568s) [noise=true , address=145.79.227.29:4951   , piece_id=Z3EFW2I2ZVWYYJHHGRODEOWHIZPNUPRN4ZYGSJDLSDSKRERIGJIQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.300668383 (587h19m12.171751428s) [noise=true , address=145.79.227.21:4481   , piece_id=LMJKSWB2QVI7KVQXTEUOTS7CRTSW3OH3JOAHX7QLSMHHVOVYV6UA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387586508 (587h19m12.258669553s) [noise=true , address=145.79.227.30:4785   , piece_id=2BYGPRYW2RPEXPZRL4K6LF65OMZVIW5OALYE5WWXLGPJYYUBIE3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: <nil>
Dec 30 02:48:12.387605635 (587h19m12.25868867s) [seg_pos={0 0}, upload=514865]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:48:12.387772603 (587h19m12.258855638s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.387778865 (587h19m12.25886191s) [noise=true , address=77.90.31.13:4646     , piece_id=LMCNMIVWNMW2XR5OD3ZGI7O36XQWTJYRKNMSGX6VBPEEKQ53PI3A, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.387991029 (587h19m12.259074064s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60988->64.49.13.169:4713: use of closed network connection
Dec 30 02:48:12.388000327 (587h19m12.259083362s) [noise=true , address=64.49.13.169:4713    , piece_id=QKXVXMV2IES4HKZ4VYP637R2NGNRKKT2TVTDBV76TXBJYDFQELNQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388145934 (587h19m12.259228979s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388156765 (587h19m12.2592398s) [noise=true , address=77.90.31.14:4678     , piece_id=7ZQ6SQQEEAJUWEXA5JF5DQFNHG27VXOE7UCVFUIRWB5NQDR4TMVA, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388257667 (587h19m12.259340702s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:12.388264229 (587h19m12.259347264s) [noise=true , address=145.79.227.33:4877   , piece_id=VHDLVPJ72WJDPBJI5FH75WM7ZOEPLPZUKGAMQLCQMC2I47OBH4PQ, seg_pos={0 0}, upload=514865]: Long tail cancellation due to optimal threshold
Dec 30 02:48:12.388279107 (587h19m12.259362142s) [seg_pos={0 0}, upload=514865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:48:12.388347377 (587h19m12.259430412s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:12.388354431 (587h19m12.259437476s) [upload=514865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:12.676118889 (587h19m12.547201934s) [upload=514865]: Done uploading segments...
Dec 30 02:48:12.676135632 (587h19m12.547218667s) [upload=514865]: Upload finished. err: <nil>
Duration: 1m0.003319807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XXX]
duration: 
30.004607848s
trace: 
https://jaeger.dev.storj.io/trace/6f3cd5d32d1bb8d1
FAIL (2025-12-30 02:49:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ) context deadline exceeded
======LOGS======
 30 02:48:34.132668071 (587h19m34.003751117s) [upload=514869]: Starting upload
Dec 30 02:48:34.132732373 (587h19m34.003815408s) [upload=514869]: Done starting upload
Dec 30 02:48:34.132856309 (587h19m34.003939355s) [upload=514869]: Uploading segments...
Dec 30 02:48:34.132958214 (587h19m34.004041259s) [upload=514869]: Got next segment. Inline: false
Dec 30 02:48:34.133005884 (587h19m34.004088929s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:48:34.133012126 (587h19m34.004095171s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:48:34.376429361 (587h19m34.247512396s) [seg_pos={0 0}, upload=514869]: Begin upload segment...
Dec 30 02:48:34.376473975 (587h19m34.24755701s) [seg_pos={0 0}, upload=514869]: Done begin upload segment.
Dec 30 02:48:34.376497621 (587h19m34.247580656s) [noise=true , address=45.91.116.6:4975     , piece_id=OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.376550021 (587h19m34.247633056s) [noise=true , address=145.79.227.17:4971   , piece_id=NHFJV7CX4WCYPCI4KQGN7IWS7IVCY4CZZ5XV73FQJCRODYWYZTCQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377016358 (587h19m34.248099404s) [noise=true , address=77.90.31.15:4697     , piece_id=2US5STRDRCBRAXTOXOSYOQNNKW2CRG7YXMJ7V3C2OLJC2MFGBKBQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377609598 (587h19m34.248692633s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.377963191 (587h19m34.249046226s) [noise=true , address=145.79.227.8:4329    , piece_id=7AQ6XYA626UTWA4I5OSQSPN676AQC24S7CU7DEXGFOPAJNHW4BJA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378353735 (587h19m34.24943677s) [noise=true , address=145.79.227.18:4006   , piece_id=TOK56L3T7IWS7MNMM3IPVGPBRCCVU4IRWG47OIUB3RSLNYSXJ2SQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378638126 (587h19m34.249721161s) [noise=true , address=145.79.227.38:4977   , piece_id=SWYUWZUZ3RKHAXDLPFQFHPTCDBPNRQPQVUJOSBARAZUZHM5LUYOQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.378920343 (587h19m34.250003378s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.379235735 (587h19m34.25031887s) [noise=true , address=145.79.227.28:4727   , piece_id=DELXWBPZUAK5ITQVN4TQTMW237YSYQFCIWFB7WULUOVSGFGTJU7A, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.380116522 (587h19m34.251199557s) [noise=true , address=45.91.116.8:4046     , piece_id=JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.381401599 (587h19m34.252484644s) [noise=true , address=145.79.227.7:4294    , piece_id=2Z6XZNHCCOXORXPQWBBMVKVMA75HA7AFWW6534FXBKO43MXFYX5Q, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.382511973 (587h19m34.253595008s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.383619060 (587h19m34.254702096s) [noise=true , address=145.79.227.42:4122   , piece_id=HG6O4FI2A7QJG7HU33KVWN4HFV5Z7HBGSQGYIKNKWZKTIOGQUIPQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.384847931 (587h19m34.255930966s) [noise=true , address=145.79.227.11:4729   , piece_id=PCEGG75NP63ODMSMNDQT7JRBWSCDOWDJG7YDCGJ57DUT4YD75DCA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.385989285 (587h19m34.25707232s) [noise=true , address=145.79.227.37:4565   , piece_id=7XKP6F5R5DRBGZQ5634EJZRVBLTMRBAGJ7DTKODVBPDIYVXSK4FQ, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:34.387433374 (587h19m34.258516409s) [noise=true , address=145.79.227.14:4837   , piece_id=6LWF73KBSOVQBIN6X3ZDHF5LVBKIE4NSCLVRIPCHEZDKFZMDNRPA, seg_pos={0 0}, upload=514869]: Uploading piece...
Dec 30 02:48:36.089013448 (587h19m35.960096483s) [upload=514869]: Next returned nil segment
Dec 30 02:48:36.089043345 (587h19m35.960126411s) [upload=514869]: Waiting for error group managing segments...
Dec 30 02:48:36.364094246 (587h19m36.235177281s) [noise=true , address=145.79.227.17:4971   , piece_id=NHFJV7CX4WCYPCI4KQGN7IWS7IVCY4CZZ5XV73FQJCRODYWYZTCQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.366015063 (587h19m36.237098108s) [noise=true , address=145.79.227.14:4837   , piece_id=6LWF73KBSOVQBIN6X3ZDHF5LVBKIE4NSCLVRIPCHEZDKFZMDNRPA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.368414632 (587h19m36.239497667s) [noise=true , address=145.79.227.42:4122   , piece_id=HG6O4FI2A7QJG7HU33KVWN4HFV5Z7HBGSQGYIKNKWZKTIOGQUIPQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.378961736 (587h19m36.250044771s) [noise=true , address=145.79.227.8:4329    , piece_id=7AQ6XYA626UTWA4I5OSQSPN676AQC24S7CU7DEXGFOPAJNHW4BJA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.385773736 (587h19m36.256856771s) [noise=true , address=145.79.227.11:4729   , piece_id=PCEGG75NP63ODMSMNDQT7JRBWSCDOWDJG7YDCGJ57DUT4YD75DCA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.389077387 (587h19m36.260160422s) [noise=true , address=145.79.227.28:4727   , piece_id=DELXWBPZUAK5ITQVN4TQTMW237YSYQFCIWFB7WULUOVSGFGTJU7A, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.392734992 (587h19m36.263818027s) [noise=true , address=145.79.227.37:4565   , piece_id=7XKP6F5R5DRBGZQ5634EJZRVBLTMRBAGJ7DTKODVBPDIYVXSK4FQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.408991477 (587h19m36.280074532s) [noise=true , address=145.79.227.7:4294    , piece_id=2Z6XZNHCCOXORXPQWBBMVKVMA75HA7AFWW6534FXBKO43MXFYX5Q, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.415968873 (587h19m36.287051918s) [noise=true , address=145.79.227.18:4006   , piece_id=TOK56L3T7IWS7MNMM3IPVGPBRCCVU4IRWG47OIUB3RSLNYSXJ2SQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.521606163 (587h19m36.392689198s) [noise=true , address=145.79.227.38:4977   , piece_id=SWYUWZUZ3RKHAXDLPFQFHPTCDBPNRQPQVUJOSBARAZUZHM5LUYOQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.675426310 (587h19m36.546509355s) [noise=true , address=77.90.31.15:4697     , piece_id=2US5STRDRCBRAXTOXOSYOQNNKW2CRG7YXMJ7V3C2OLJC2MFGBKBQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.727408643 (587h19m36.598491688s) [noise=true , address=45.91.116.8:4046     , piece_id=JM2A5PD6SKTYZVHEJG2VYT2PWURRIHV2DANXCWSZORCP6EDIP5IA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.821633622 (587h19m36.692716657s) [noise=true , address=45.91.116.6:4975     , piece_id=OJNDXWPQP3UB6REVQJEKXZEU4WTBNRA3L33RM57XLFP3MQBXKMKQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: <nil>
Dec 30 02:48:36.821650634 (587h19m36.692733669s) [seg_pos={0 0}, upload=514869]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:48:36.821841658 (587h19m36.692924693s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:36.821848210 (587h19m36.692931245s) [noise=true , address=145.79.227.49:4339   , piece_id=OWK7X732NS2XXXCQTRX3WEWINUPWPVPPYKLXSES3YEYWM44LUBQA, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.821964291 (587h19m36.693047326s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->77.90.31.11:4556: use of closed network connection
Dec 30 02:48:36.821969161 (587h19m36.693052196s) [noise=true , address=77.90.31.11:4556     , piece_id=NFKLLOZ6HTCQNGW6N34GIDOXI2WF5ZJHUTYPRS64UTVGWLC3M4LQ, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.822204739 (587h19m36.693287784s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Dec 30 02:48:36.822240267 (587h19m36.693323302s) [noise=true , address=77.90.31.19:4842     , piece_id=PMKBOJL7GLXIN4RVMEXXORLHQST6ZUTR263GL5G5XJ7MHOS76Y5A, seg_pos={0 0}, upload=514869]: Long tail cancellation due to optimal threshold
Dec 30 02:48:36.822266086 (587h19m36.693349121s) [seg_pos={0 0}, upload=514869]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:48:36.822369593 (587h19m36.693452628s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:48:36.822374983 (587h19m36.693458018s) [upload=514869]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:48:37.116690878 (587h19m36.987773923s) [upload=514869]: Done uploading segments...
Dec 30 02:48:37.116712188 (587h19m36.987795223s) [upload=514869]: Upload finished. err: <nil>
Duration: 30.004634117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [...........X......X...................................X...X.]
duration: 
3.573999192s
trace: 
https://jaeger.dev.storj.io/trace/4ccd4b8482af458e
FAIL (2025-12-30 02:48:40 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.574026723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [....X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......XX]
duration: 
30.002829324s
trace: 
https://jaeger.dev.storj.io/trace/7d184635e89e174d
FAIL (2025-12-30 02:48:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 02:47:56.302181517 (587h18m56.173264562s) [upload=514859]: Starting upload
Dec 30 02:47:56.302250047 (587h18m56.173333092s) [upload=514859]: Done starting upload
Dec 30 02:47:56.302303679 (587h18m56.173386724s) [upload=514859]: Uploading segments...
Dec 30 02:47:56.302425692 (587h18m56.173508727s) [upload=514859]: Got next segment. Inline: false
Dec 30 02:47:56.302453785 (587h18m56.17353682s) [upload=514859]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:47:56.302458674 (587h18m56.173541709s) [upload=514859]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:47:56.549647931 (587h18m56.420730967s) [seg_pos={0 0}, upload=514859]: Begin upload segment...
Dec 30 02:47:56.549705792 (587h18m56.420788837s) [seg_pos={0 0}, upload=514859]: Done begin upload segment.
Dec 30 02:47:56.549750467 (587h18m56.420833512s) [noise=true , address=145.79.227.30:4761   , piece_id=H7PPBJ2VJRGSXV2QUV2O2APRCWS75MTOAAWUASUYAELUY5MNSAHA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.549842161 (587h18m56.420925206s) [noise=true , address=45.91.116.11:4117    , piece_id=N4P4YBD2DKPEOR5G4W7JBCDJOXQ7Y7IIPPZJ44MFHVTWTHWPPOHQ, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.550640451 (587h18m56.421723496s) [noise=true , address=145.79.227.50:4373   , piece_id=SWC2OAUCMC7TQMNQ46N3VS3UHHBZUTKPO5A2WCP4C6ZI5I4WW4WA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.550989626 (587h18m56.422072661s) [noise=true , address=45.91.116.6:4975     , piece_id=M7UCTXAQGDTYNWBAYQFBKAGJKFOXKEB5ZPHGHQ5NTMEFL7VOCY7A, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.551239192 (587h18m56.422322237s) [noise=true , address=145.79.227.48:4302   , piece_id=XKAKXHTDGB46EBOSVU3CS7XMOMVKSHOO3OKUQQIUVTUSQRGXXIFA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.551304485 (587h18m56.42238753s) [noise=true , address=145.79.227.15:4875   , piece_id=NSW4P7IAMEE3HH5KGT2OVX62MKDFGNT62JC3QALT5CLJSAFWKLXA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.551588256 (587h18m56.422671302s) [noise=true , address=145.79.227.52:4445   , piece_id=DW26F4X37KCUUFUAEOLO4XQ5LGKZ5XVQ6SDV7FO7FIJX7LLCCVGQ, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.551619245 (587h18m56.42270228s) [noise=true , address=145.79.227.16:4912   , piece_id=XJ3WQQ6QZNF3BEC25Z65R2Q752RGYVBYRBEGAKDGZUJ66WCM6KFQ, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.552140858 (587h18m56.423223903s) [noise=true , address=77.90.31.15:4697     , piece_id=6XQDV4NVPTCBZGST3F772KUX6EWPCWQD5LC545OXTWZAVSKKULHA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.552589292 (587h18m56.423672327s) [noise=true , address=145.79.227.38:4979   , piece_id=F3ADN7S3EAYHSI6MIN6BWHBUQGRAUEW77WO5PO5VPOYSEBMXLRWQ, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.552996347 (587h18m56.424079392s) [noise=true , address=45.91.116.5:4298     , piece_id=QJVVUK5SVQ2TYTIFAT5FEQJ3LKIJQYQGQBDH323LDBZBAFSXVVDA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.553141073 (587h18m56.424224108s) [noise=true , address=45.91.116.10:4082    , piece_id=CSO527WNFMB5Q6EA3QG5EOJ7FUAZRRI26254FDKSG6DVIHZ6MFPA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.553471192 (587h18m56.424554237s) [noise=true , address=77.90.31.14:4663     , piece_id=HJNIV6YPKCXT7J3A7VJNW6QGEV5R7BOSU6HLFSKOMS3ZBHZP7YGA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.553688425 (587h18m56.42477146s) [noise=true , address=77.90.31.19:4842     , piece_id=OLFOALFKL4OS7K2ADN4JYXRFP7I4BNU55M2W3TAAEO6OKZBG5Q5Q, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.554709880 (587h18m56.425792925s) [noise=true , address=77.90.31.21:4878     , piece_id=Y424Q7UUXQDFNTOSOO676ZW2LCSVULI4GOBH6UQYUO6Q264J7O6Q, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:56.555088701 (587h18m56.426171746s) [noise=true , address=145.79.227.14:4838   , piece_id=BKMZD4F2ZY7P4W5VVVKSGWZXG73ADWG5GV6CVFTR6O5P5WVOVFFA, seg_pos={0 0}, upload=514859]: Uploading piece...
Dec 30 02:47:58.246780117 (587h18m58.117863172s) [upload=514859]: Next returned nil segment
Dec 30 02:47:58.246795486 (587h18m58.117878521s) [upload=514859]: Waiting for error group managing segments...
Dec 30 02:47:58.531430726 (587h18m58.402513771s) [noise=true , address=145.79.227.15:4875   , piece_id=NSW4P7IAMEE3HH5KGT2OVX62MKDFGNT62JC3QALT5CLJSAFWKLXA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.533042826 (587h18m58.404125861s) [noise=true , address=145.79.227.52:4445   , piece_id=DW26F4X37KCUUFUAEOLO4XQ5LGKZ5XVQ6SDV7FO7FIJX7LLCCVGQ, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.540960521 (587h18m58.412043567s) [noise=true , address=145.79.227.16:4912   , piece_id=XJ3WQQ6QZNF3BEC25Z65R2Q752RGYVBYRBEGAKDGZUJ66WCM6KFQ, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.561492528 (587h18m58.432575573s) [noise=true , address=145.79.227.38:4979   , piece_id=F3ADN7S3EAYHSI6MIN6BWHBUQGRAUEW77WO5PO5VPOYSEBMXLRWQ, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.571136362 (587h18m58.442219397s) [noise=true , address=145.79.227.50:4373   , piece_id=SWC2OAUCMC7TQMNQ46N3VS3UHHBZUTKPO5A2WCP4C6ZI5I4WW4WA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.589644273 (587h18m58.460727328s) [noise=true , address=145.79.227.14:4838   , piece_id=BKMZD4F2ZY7P4W5VVVKSGWZXG73ADWG5GV6CVFTR6O5P5WVOVFFA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.743288707 (587h18m58.614371742s) [noise=true , address=77.90.31.21:4878     , piece_id=Y424Q7UUXQDFNTOSOO676ZW2LCSVULI4GOBH6UQYUO6Q264J7O6Q, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.782822305 (587h18m58.65390534s) [noise=true , address=45.91.116.11:4117    , piece_id=N4P4YBD2DKPEOR5G4W7JBCDJOXQ7Y7IIPPZJ44MFHVTWTHWPPOHQ, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.850654498 (587h18m58.721737533s) [noise=true , address=45.91.116.6:4975     , piece_id=M7UCTXAQGDTYNWBAYQFBKAGJKFOXKEB5ZPHGHQ5NTMEFL7VOCY7A, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:58.980159636 (587h18m58.851242671s) [noise=true , address=45.91.116.10:4082    , piece_id=CSO527WNFMB5Q6EA3QG5EOJ7FUAZRRI26254FDKSG6DVIHZ6MFPA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:59.235253464 (587h18m59.106336499s) [noise=true , address=77.90.31.15:4697     , piece_id=6XQDV4NVPTCBZGST3F772KUX6EWPCWQD5LC545OXTWZAVSKKULHA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:59.347568280 (587h18m59.218651315s) [noise=true , address=77.90.31.14:4663     , piece_id=HJNIV6YPKCXT7J3A7VJNW6QGEV5R7BOSU6HLFSKOMS3ZBHZP7YGA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:59.672555005 (587h18m59.54363804s) [noise=true , address=77.90.31.19:4842     , piece_id=OLFOALFKL4OS7K2ADN4JYXRFP7I4BNU55M2W3TAAEO6OKZBG5Q5Q, seg_pos={0 0}, upload=514859]: Done uploading piece. err: <nil>
Dec 30 02:47:59.672586906 (587h18m59.543669951s) [seg_pos={0 0}, upload=514859]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:47:59.672806423 (587h18m59.543889458s) [noise=true , address=45.91.116.5:4298     , piece_id=QJVVUK5SVQ2TYTIFAT5FEQJ3LKIJQYQGQBDH323LDBZBAFSXVVDA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 02:47:59.672813277 (587h18m59.543896322s) [noise=true , address=45.91.116.5:4298     , piece_id=QJVVUK5SVQ2TYTIFAT5FEQJ3LKIJQYQGQBDH323LDBZBAFSXVVDA, seg_pos={0 0}, upload=514859]: Long tail cancellation due to optimal threshold
Dec 30 02:47:59.672974033 (587h18m59.544057078s) [noise=true , address=145.79.227.30:4761   , piece_id=H7PPBJ2VJRGSXV2QUV2O2APRCWS75MTOAAWUASUYAELUY5MNSAHA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 30 02:47:59.673004310 (587h18m59.544087345s) [noise=true , address=145.79.227.30:4761   , piece_id=H7PPBJ2VJRGSXV2QUV2O2APRCWS75MTOAAWUASUYAELUY5MNSAHA, seg_pos={0 0}, upload=514859]: Long tail cancellation due to optimal threshold
Dec 30 02:47:59.673053535 (587h18m59.54413658s) [noise=true , address=145.79.227.48:4302   , piece_id=XKAKXHTDGB46EBOSVU3CS7XMOMVKSHOO3OKUQQIUVTUSQRGXXIFA, seg_pos={0 0}, upload=514859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:40062->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:40062->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->145.79.227.48:4302: use of closed network connection
Dec 30 02:47:59.673061379 (587h18m59.544144414s) [noise=true , address=145.79.227.48:4302   , piece_id=XKAKXHTDGB46EBOSVU3CS7XMOMVKSHOO3OKUQQIUVTUSQRGXXIFA, seg_pos={0 0}, upload=514859]: Long tail cancellation due to optimal threshold
Dec 30 02:47:59.673084733 (587h18m59.544167768s) [seg_pos={0 0}, upload=514859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:47:59.673155839 (587h18m59.544238884s) [upload=514859]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:47:59.673160929 (587h18m59.544243964s) [upload=514859]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:47:59.957368664 (587h18m59.828451719s) [upload=514859]: Done uploading segments...
Dec 30 02:47:59.957395205 (587h18m59.82847825s) [upload=514859]: Upload finished. err: <nil>
Duration: 30.002855964s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [............X......X...................................X...X]
duration: 
5.001740956s
trace: 
https://jaeger.dev.storj.io/trace/4ccd4b8482af458e
FAIL (2025-12-30 02:48:05 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001766916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............................X.............X................]
duration: 
15.349250259s
trace: 
https://jaeger.dev.storj.io/trace/64e31d9fb30f9193
FAIL (2025-12-30 02:48:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, Piece ID: PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA) context deadline exceeded; piecestore: (Node ID: 12ZNCRKd7xMFNHfCJBJ38yfyUmczhYwJetKtiQxX9SDrwxRhzTU, Piece ID: VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA) context deadline exceeded; piecestore: (Node ID: 12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, Piece ID: 2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA) context deadline exceeded
======LOGS======
HRMYNPQZZ2A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.247734605 (587h4m42.11881765s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.248134877 (587h4m42.119217922s) [noise=true , address=45.91.116.14:4212    , piece_id=PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.248843948 (587h4m42.119926993s) [noise=true , address=145.79.227.40:4072   , piece_id=CXD4SA6GCLT5PF2LIWPTZ6SGB26MXVW4L7DYAO5XEX4RTQNR2RQA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.249476452 (587h4m42.120559487s) [noise=true , address=145.79.227.29:4967   , piece_id=6GLQROE5CKKU4HYBLDXULBKY7GSQR3K3BNLUGNP3Q244T226U46A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.250154953 (587h4m42.121237999s) [noise=true , address=145.79.227.50:4406   , piece_id=AUQ42O4S3DKDD2MKF57OS4Y2LM6NT3RF7QGN3QYCJKVXYKXMDGHA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.251492680 (587h4m42.122575715s) [noise=true , address=45.91.116.12:4251    , piece_id=VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.252137288 (587h4m42.123220323s) [noise=true , address=77.90.31.17:4769     , piece_id=2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.253274743 (587h4m42.124357779s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.254335895 (587h4m42.12541894s) [noise=true , address=145.79.227.31:4801   , piece_id=HP33YDMBWOUTJ5EV46CYF6I5DYRU3NBBBCCJPMIND4BGERBZCBXA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.255263290 (587h4m42.126346325s) [noise=true , address=145.79.227.30:4792   , piece_id=NGPB4ETVT3T27F4NVVXAPFCZJDFLKCDZLNB5TPSQVXTU6JOLONHA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.256371450 (587h4m42.127454485s) [noise=true , address=45.91.116.5:4321     , piece_id=QLRY6QPHREOCP3Y4JLEO4GG43US2RXQIQS3J5QWK2B3UY5VKQOAA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.256731046 (587h4m42.127814091s) [noise=true , address=145.79.227.35:4512   , piece_id=VGZSUBQA4O44NZ3E7LX7VZJGWPASYRCMYWXMMZ63CNUQIPP56QKA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.258562643 (587h4m42.129645678s) [noise=true , address=145.79.227.14:4845   , piece_id=FCOU6BUEKBVMJFBEZCR3WRAWBAU3MGLR2HPYNJLGW67GD7GARW6A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.259130144 (587h4m42.130213179s) [noise=true , address=145.79.227.44:4211   , piece_id=7RYUYPKG6SEH2K4YL7CBNGK54AWUXRNJKUDMBM4MV45ZTCSJJV7A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:43.629886831 (587h4m43.500969866s) [upload=514659]: Next returned nil segment
Dec 30 02:33:43.629901209 (587h4m43.500984254s) [upload=514659]: Waiting for error group managing segments...
Dec 30 02:33:44.106697785 (587h4m43.97778083s) [noise=true , address=64.49.13.27:4959     , piece_id=FBWYKY2DMAWOSRYRGMVKK3MRCC3KXCBHFUUNRWD5UHRMYNPQZZ2A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.120802703 (587h4m43.991885748s) [noise=true , address=145.79.227.26:4687   , piece_id=6S32MBKUUTBX7OOYKXWMQKA2CPA57UVD2FUOFV4CM3JBHAWSUA7Q, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.150792138 (587h4m44.021875183s) [noise=true , address=145.79.227.50:4406   , piece_id=AUQ42O4S3DKDD2MKF57OS4Y2LM6NT3RF7QGN3QYCJKVXYKXMDGHA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.219759101 (587h4m44.090842137s) [noise=true , address=145.79.227.31:4801   , piece_id=HP33YDMBWOUTJ5EV46CYF6I5DYRU3NBBBCCJPMIND4BGERBZCBXA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.346226767 (587h4m44.217309802s) [noise=true , address=145.79.227.14:4845   , piece_id=FCOU6BUEKBVMJFBEZCR3WRAWBAU3MGLR2HPYNJLGW67GD7GARW6A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.416827681 (587h4m44.287910726s) [noise=true , address=45.91.116.14:4212    , piece_id=PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.435658097 (587h4m44.306741132s) [noise=true , address=45.91.116.5:4321     , piece_id=QLRY6QPHREOCP3Y4JLEO4GG43US2RXQIQS3J5QWK2B3UY5VKQOAA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.470304881 (587h4m44.341387916s) [noise=true , address=45.91.116.12:4251    , piece_id=VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.742645801 (587h4m44.613728846s) [noise=true , address=77.90.31.17:4769     , piece_id=2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.001291913 (587h4m44.872374968s) [noise=true , address=145.79.227.40:4072   , piece_id=CXD4SA6GCLT5PF2LIWPTZ6SGB26MXVW4L7DYAO5XEX4RTQNR2RQA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.028105611 (587h4m44.899188656s) [noise=true , address=145.79.227.30:4792   , piece_id=NGPB4ETVT3T27F4NVVXAPFCZJDFLKCDZLNB5TPSQVXTU6JOLONHA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.053144649 (587h4m44.924227694s) [noise=true , address=145.79.227.44:4211   , piece_id=7RYUYPKG6SEH2K4YL7CBNGK54AWUXRNJKUDMBM4MV45ZTCSJJV7A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.093008306 (587h4m44.964091341s) [noise=true , address=145.79.227.35:4512   , piece_id=VGZSUBQA4O44NZ3E7LX7VZJGWPASYRCMYWXMMZ63CNUQIPP56QKA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.235172189 (587h4m45.106255224s) [noise=true , address=145.79.227.29:4967   , piece_id=6GLQROE5CKKU4HYBLDXULBKY7GSQR3K3BNLUGNP3Q244T226U46A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.235193228 (587h4m45.106276263s) [seg_pos={0 0}, upload=514659]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:33:45.235447733 (587h4m45.106530788s) [noise=true , address=77.90.31.11:4582     , piece_id=53G52XQXHBAAAD6FEZUVOAKRMQADO7ODQB2GNJFOHHYM4ALJUBXA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection; write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection
Dec 30 02:33:45.235460017 (587h4m45.106543052s) [noise=true , address=77.90.31.11:4582     , piece_id=53G52XQXHBAAAD6FEZUVOAKRMQADO7ODQB2GNJFOHHYM4ALJUBXA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235465226 (587h4m45.106548261s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG): protocol: write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection; write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection
Dec 30 02:33:45.235473973 (587h4m45.106557008s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235655508 (587h4m45.106738543s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: expected piece hash; context canceled; EOF
Dec 30 02:33:45.235664416 (587h4m45.106747471s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235778813 (587h4m45.106861859s) [noise=true , address=45.91.116.9:4280     , piece_id=GU64PSW7MAS6EJQ7ELRDUXDUXCPB5OYX7Q2NELUK5IM33QQAHTDQ, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: read tcp 188.166.225.170:37200->45.91.116.9:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 02:33:45.235785917 (587h4m45.106868962s) [noise=true , address=45.91.116.9:4280     , piece_id=GU64PSW7MAS6EJQ7ELRDUXDUXCPB5OYX7Q2NELUK5IM33QQAHTDQ, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235800013 (587h4m45.106883058s) [seg_pos={0 0}, upload=514659]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:33:45.235869375 (587h4m45.10695241s) [upload=514659]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:33:45.235873674 (587h4m45.106956709s) [upload=514659]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:33:45.520374428 (587h4m45.391457473s) [upload=514659]: Done uploading segments...
Dec 30 02:33:45.520402863 (587h4m45.391485908s) [upload=514659]: Upload finished. err: <nil>
Duration: 15.349309891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X....X..X.XXX..X.XXXX..X......X........X...X..XX..X.X......X]
duration: 
30.004835191s
trace: 
https://jaeger.dev.storj.io/trace/4da26136f700c826
FAIL (2025-12-30 02:47:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: SG7I63AODATUUSBKBUTVLHTXMHICPM24STEAVYHSY5HIOZG4QASQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: BE2YOHSU5TVPH2RLJMSSIT3KBYGY3DHYCIPOFCTYZA4TZHJ6TREA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 7EK2U4XWHOEZNKRREAPFDHKHUUZVAR3KLVI63BEHWIWROHCCT7DQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: Q6VY43LTOGU4WFKCOLFLE6CRD4XPYFTAI4YCVG5LR5JMCX5HZMVA) context deadline exceeded
======LOGS======
18m22.508704112s) [upload=514852]: Done starting upload
Dec 30 02:47:22.637676413 (587h18m22.508759458s) [upload=514852]: Uploading segments...
Dec 30 02:47:22.638031468 (587h18m22.509114503s) [upload=514852]: Got next segment. Inline: false
Dec 30 02:47:22.638178568 (587h18m22.509261613s) [upload=514852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 02:47:22.638190381 (587h18m22.509273447s) [upload=514852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 02:47:22.883025445 (587h18m22.75410849s) [seg_pos={0 0}, upload=514852]: Begin upload segment...
Dec 30 02:47:22.883081052 (587h18m22.754164107s) [seg_pos={0 0}, upload=514852]: Done begin upload segment.
Dec 30 02:47:22.883130535 (587h18m22.754213571s) [noise=true , address=145.79.227.42:4121   , piece_id=6HIOUDSTRZRV3PACGPGN77CDC2KABDXRB4VCALVZ45KO6DMOZ6HA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.883148269 (587h18m22.754231304s) [noise=true , address=145.79.227.43:4159   , piece_id=C544HQQHRY2R7Q76VV5USCDKT4W6WYOJ4R3IZYPNHR2ULE3GVGDA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.883759433 (587h18m22.754842508s) [noise=true , address=145.79.227.8:4331    , piece_id=3KAUL76GVYGOY5NO4W2JJKDS3HSQ3BMFXVJNYVOLMWY66OFACZ6Q, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.884457522 (587h18m22.755540557s) [noise=true , address=77.90.31.16:4735     , piece_id=MT5Y23L6GNBDEFTYKKXFBKAHXUG3YIIF574K3JEJZ6LBZYGHKL6Q, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.884872633 (587h18m22.755955678s) [noise=true , address=77.90.31.11:4553     , piece_id=FGJQOZHXDJTVJ2FZDTUR4YZENRO4NFMCVJZRVI37VRW3VAOGFA7A, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.885266813 (587h18m22.756349858s) [noise=true , address=145.79.227.16:4912   , piece_id=IRDH7BN6Y5D4VNAXPQAFTJXQ62UYLKXTDFL4OI7KB4PPWY5ZZJOA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.885766285 (587h18m22.75684933s) [noise=true , address=145.79.227.15:4873   , piece_id=LJZBGG5KAHCWOUT2AK3UDPQKHGKRYDHIHNPFLPNPAIC3ZLMY5MKQ, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.886413016 (587h18m22.757496051s) [noise=true , address=45.91.116.11:4119    , piece_id=BE2YOHSU5TVPH2RLJMSSIT3KBYGY3DHYCIPOFCTYZA4TZHJ6TREA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.887760482 (587h18m22.758843517s) [noise=true , address=45.91.116.6:4973     , piece_id=7EK2U4XWHOEZNKRREAPFDHKHUUZVAR3KLVI63BEHWIWROHCCT7DQ, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.887964811 (587h18m22.759047856s) [noise=true , address=145.79.227.5:4222    , piece_id=4MQC6NLPMN5I7ULOJF6NAPQACMCPUAZUSQZ3KKARXM33E4ICR7JA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.889340831 (587h18m22.760423866s) [noise=true , address=77.90.31.15:4697     , piece_id=Q6VY43LTOGU4WFKCOLFLE6CRD4XPYFTAI4YCVG5LR5JMCX5HZMVA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.889603120 (587h18m22.760686156s) [noise=true , address=77.90.31.19:4842     , piece_id=SG7I63AODATUUSBKBUTVLHTXMHICPM24STEAVYHSY5HIOZG4QASQ, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.893994933 (587h18m22.765077968s) [noise=true , address=145.79.227.32:4836   , piece_id=Z7CX42WNDHFCCFDYZRXMGWRSJMP2B6O5UUZA6QYYBONV6WZAM6VQ, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.899865021 (587h18m22.770948066s) [noise=true , address=145.79.227.44:4194   , piece_id=3AVA2HE3ZJCHKVOQCWFHI5E6SBZCEAEQXKUD2EM2MFCQVGPBE77A, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.902233100 (587h18m22.773316135s) [noise=true , address=145.79.227.51:4410   , piece_id=GET3OE3O6IVEBEUM3WJELQKF6DA4B2SFSRV7MIH465RKGPNUAQIA, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:22.903352291 (587h18m22.774435326s) [noise=true , address=145.79.227.11:4731   , piece_id=K6MUD2HF6V2KSGZSZMHO2EBHQUW34IXOFMOJSQN5HJ4W7WABTMOQ, seg_pos={0 0}, upload=514852]: Uploading piece...
Dec 30 02:47:24.593396619 (587h18m24.464479664s) [upload=514852]: Next returned nil segment
Dec 30 02:47:24.593433179 (587h18m24.464516224s) [upload=514852]: Waiting for error group managing segments...
Dec 30 02:47:24.881551992 (587h18m24.752635027s) [noise=true , address=145.79.227.32:4836   , piece_id=Z7CX42WNDHFCCFDYZRXMGWRSJMP2B6O5UUZA6QYYBONV6WZAM6VQ, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.882147015 (587h18m24.75323005s) [noise=true , address=145.79.227.11:4731   , piece_id=K6MUD2HF6V2KSGZSZMHO2EBHQUW34IXOFMOJSQN5HJ4W7WABTMOQ, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.888789894 (587h18m24.759872929s) [noise=true , address=145.79.227.15:4873   , piece_id=LJZBGG5KAHCWOUT2AK3UDPQKHGKRYDHIHNPFLPNPAIC3ZLMY5MKQ, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.889362434 (587h18m24.760445469s) [noise=true , address=145.79.227.8:4331    , piece_id=3KAUL76GVYGOY5NO4W2JJKDS3HSQ3BMFXVJNYVOLMWY66OFACZ6Q, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.895118214 (587h18m24.766201259s) [noise=true , address=145.79.227.42:4121   , piece_id=6HIOUDSTRZRV3PACGPGN77CDC2KABDXRB4VCALVZ45KO6DMOZ6HA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.896037434 (587h18m24.76712047s) [noise=true , address=145.79.227.5:4222    , piece_id=4MQC6NLPMN5I7ULOJF6NAPQACMCPUAZUSQZ3KKARXM33E4ICR7JA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:24.901413371 (587h18m24.772496406s) [noise=true , address=145.79.227.16:4912   , piece_id=IRDH7BN6Y5D4VNAXPQAFTJXQ62UYLKXTDFL4OI7KB4PPWY5ZZJOA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:25.132228108 (587h18m25.003311173s) [noise=true , address=77.90.31.15:4697     , piece_id=Q6VY43LTOGU4WFKCOLFLE6CRD4XPYFTAI4YCVG5LR5JMCX5HZMVA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:25.170940031 (587h18m25.042023066s) [noise=true , address=77.90.31.11:4553     , piece_id=FGJQOZHXDJTVJ2FZDTUR4YZENRO4NFMCVJZRVI37VRW3VAOGFA7A, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:25.209101708 (587h18m25.080184763s) [noise=true , address=45.91.116.6:4973     , piece_id=7EK2U4XWHOEZNKRREAPFDHKHUUZVAR3KLVI63BEHWIWROHCCT7DQ, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:25.446185933 (587h18m25.317268968s) [noise=true , address=45.91.116.11:4119    , piece_id=BE2YOHSU5TVPH2RLJMSSIT3KBYGY3DHYCIPOFCTYZA4TZHJ6TREA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:26.184925319 (587h18m26.056008364s) [noise=true , address=77.90.31.19:4842     , piece_id=SG7I63AODATUUSBKBUTVLHTXMHICPM24STEAVYHSY5HIOZG4QASQ, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:26.253194774 (587h18m26.124277809s) [noise=true , address=145.79.227.44:4194   , piece_id=3AVA2HE3ZJCHKVOQCWFHI5E6SBZCEAEQXKUD2EM2MFCQVGPBE77A, seg_pos={0 0}, upload=514852]: Done uploading piece. err: <nil>
Dec 30 02:47:26.253215393 (587h18m26.124298428s) [seg_pos={0 0}, upload=514852]: Segment reached optimal threshold of 13 pieces.
Dec 30 02:47:26.253401117 (587h18m26.124484162s) [noise=true , address=145.79.227.51:4410   , piece_id=GET3OE3O6IVEBEUM3WJELQKF6DA4B2SFSRV7MIH465RKGPNUAQIA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39126->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39126->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39126->145.79.227.51:4410: use of closed network connection
Dec 30 02:47:26.253407148 (587h18m26.124490183s) [noise=true , address=145.79.227.51:4410   , piece_id=GET3OE3O6IVEBEUM3WJELQKF6DA4B2SFSRV7MIH465RKGPNUAQIA, seg_pos={0 0}, upload=514852]: Long tail cancellation due to optimal threshold
Dec 30 02:47:26.253531385 (587h18m26.12461442s) [noise=true , address=145.79.227.43:4159   , piece_id=C544HQQHRY2R7Q76VV5USCDKT4W6WYOJ4R3IZYPNHR2ULE3GVGDA, seg_pos={0 0}, upload=514852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 30 02:47:26.253537196 (587h18m26.124620231s) [noise=true , address=145.79.227.43:4159   , piece_id=C544HQQHRY2R7Q76VV5USCDKT4W6WYOJ4R3IZYPNHR2ULE3GVGDA, seg_pos={0 0}, upload=514852]: Long tail cancellation due to optimal threshold
Dec 30 02:47:26.253626086 (587h18m26.124709121s) [noise=true , address=77.90.31.16:4735     , piece_id=MT5Y23L6GNBDEFTYKKXFBKAHXUG3YIIF574K3JEJZ6LBZYGHKL6Q, seg_pos={0 0}, upload=514852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:37710->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 02:47:26.253641314 (587h18m26.124724349s) [noise=true , address=77.90.31.16:4735     , piece_id=MT5Y23L6GNBDEFTYKKXFBKAHXUG3YIIF574K3JEJZ6LBZYGHKL6Q, seg_pos={0 0}, upload=514852]: Long tail cancellation due to optimal threshold
Dec 30 02:47:26.253660922 (587h18m26.124743957s) [seg_pos={0 0}, upload=514852]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 02:47:26.253761293 (587h18m26.124844338s) [upload=514852]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:47:26.253766262 (587h18m26.124849297s) [upload=514852]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:47:26.546127683 (587h18m26.417210739s) [upload=514852]: Done uploading segments...
Dec 30 02:47:26.546163522 (587h18m26.417246567s) [upload=514852]: Upload finished. err: <nil>
Duration: 30.004863185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............................X.............X...............]
duration: 
45.884575837s
trace: 
https://jaeger.dev.storj.io/trace/64e31d9fb30f9193
FAIL (2025-12-30 02:47:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1RiikePNRVqnERsVeC6eFxkTpttN5bkps9fwmntzX7EsxPAW6y, Piece ID: PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA) context deadline exceeded; piecestore: (Node ID: 12ZNCRKd7xMFNHfCJBJ38yfyUmczhYwJetKtiQxX9SDrwxRhzTU, Piece ID: VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA) context deadline exceeded; piecestore: (Node ID: 12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, Piece ID: 2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA) context deadline exceeded
======LOGS======
HRMYNPQZZ2A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.247734605 (587h4m42.11881765s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.248134877 (587h4m42.119217922s) [noise=true , address=45.91.116.14:4212    , piece_id=PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.248843948 (587h4m42.119926993s) [noise=true , address=145.79.227.40:4072   , piece_id=CXD4SA6GCLT5PF2LIWPTZ6SGB26MXVW4L7DYAO5XEX4RTQNR2RQA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.249476452 (587h4m42.120559487s) [noise=true , address=145.79.227.29:4967   , piece_id=6GLQROE5CKKU4HYBLDXULBKY7GSQR3K3BNLUGNP3Q244T226U46A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.250154953 (587h4m42.121237999s) [noise=true , address=145.79.227.50:4406   , piece_id=AUQ42O4S3DKDD2MKF57OS4Y2LM6NT3RF7QGN3QYCJKVXYKXMDGHA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.251492680 (587h4m42.122575715s) [noise=true , address=45.91.116.12:4251    , piece_id=VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.252137288 (587h4m42.123220323s) [noise=true , address=77.90.31.17:4769     , piece_id=2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.253274743 (587h4m42.124357779s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.254335895 (587h4m42.12541894s) [noise=true , address=145.79.227.31:4801   , piece_id=HP33YDMBWOUTJ5EV46CYF6I5DYRU3NBBBCCJPMIND4BGERBZCBXA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.255263290 (587h4m42.126346325s) [noise=true , address=145.79.227.30:4792   , piece_id=NGPB4ETVT3T27F4NVVXAPFCZJDFLKCDZLNB5TPSQVXTU6JOLONHA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.256371450 (587h4m42.127454485s) [noise=true , address=45.91.116.5:4321     , piece_id=QLRY6QPHREOCP3Y4JLEO4GG43US2RXQIQS3J5QWK2B3UY5VKQOAA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.256731046 (587h4m42.127814091s) [noise=true , address=145.79.227.35:4512   , piece_id=VGZSUBQA4O44NZ3E7LX7VZJGWPASYRCMYWXMMZ63CNUQIPP56QKA, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.258562643 (587h4m42.129645678s) [noise=true , address=145.79.227.14:4845   , piece_id=FCOU6BUEKBVMJFBEZCR3WRAWBAU3MGLR2HPYNJLGW67GD7GARW6A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:42.259130144 (587h4m42.130213179s) [noise=true , address=145.79.227.44:4211   , piece_id=7RYUYPKG6SEH2K4YL7CBNGK54AWUXRNJKUDMBM4MV45ZTCSJJV7A, seg_pos={0 0}, upload=514659]: Uploading piece...
Dec 30 02:33:43.629886831 (587h4m43.500969866s) [upload=514659]: Next returned nil segment
Dec 30 02:33:43.629901209 (587h4m43.500984254s) [upload=514659]: Waiting for error group managing segments...
Dec 30 02:33:44.106697785 (587h4m43.97778083s) [noise=true , address=64.49.13.27:4959     , piece_id=FBWYKY2DMAWOSRYRGMVKK3MRCC3KXCBHFUUNRWD5UHRMYNPQZZ2A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.120802703 (587h4m43.991885748s) [noise=true , address=145.79.227.26:4687   , piece_id=6S32MBKUUTBX7OOYKXWMQKA2CPA57UVD2FUOFV4CM3JBHAWSUA7Q, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.150792138 (587h4m44.021875183s) [noise=true , address=145.79.227.50:4406   , piece_id=AUQ42O4S3DKDD2MKF57OS4Y2LM6NT3RF7QGN3QYCJKVXYKXMDGHA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.219759101 (587h4m44.090842137s) [noise=true , address=145.79.227.31:4801   , piece_id=HP33YDMBWOUTJ5EV46CYF6I5DYRU3NBBBCCJPMIND4BGERBZCBXA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.346226767 (587h4m44.217309802s) [noise=true , address=145.79.227.14:4845   , piece_id=FCOU6BUEKBVMJFBEZCR3WRAWBAU3MGLR2HPYNJLGW67GD7GARW6A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.416827681 (587h4m44.287910726s) [noise=true , address=45.91.116.14:4212    , piece_id=PDL7OYBP54D5EPPCIYACNIWEBOMAEB5WH3JESWKQEB22EU57WYVA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.435658097 (587h4m44.306741132s) [noise=true , address=45.91.116.5:4321     , piece_id=QLRY6QPHREOCP3Y4JLEO4GG43US2RXQIQS3J5QWK2B3UY5VKQOAA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.470304881 (587h4m44.341387916s) [noise=true , address=45.91.116.12:4251    , piece_id=VQI3EFQWFK366ZY7VYDPM3T5SP6MWPSXILR54LIVDTIJBISBNNNA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:44.742645801 (587h4m44.613728846s) [noise=true , address=77.90.31.17:4769     , piece_id=2QEDJPKKR7YEJHI5YCAHCCE2WLTUFULUJPOFNML5XKCY2H6HBWWA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.001291913 (587h4m44.872374968s) [noise=true , address=145.79.227.40:4072   , piece_id=CXD4SA6GCLT5PF2LIWPTZ6SGB26MXVW4L7DYAO5XEX4RTQNR2RQA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.028105611 (587h4m44.899188656s) [noise=true , address=145.79.227.30:4792   , piece_id=NGPB4ETVT3T27F4NVVXAPFCZJDFLKCDZLNB5TPSQVXTU6JOLONHA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.053144649 (587h4m44.924227694s) [noise=true , address=145.79.227.44:4211   , piece_id=7RYUYPKG6SEH2K4YL7CBNGK54AWUXRNJKUDMBM4MV45ZTCSJJV7A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.093008306 (587h4m44.964091341s) [noise=true , address=145.79.227.35:4512   , piece_id=VGZSUBQA4O44NZ3E7LX7VZJGWPASYRCMYWXMMZ63CNUQIPP56QKA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.235172189 (587h4m45.106255224s) [noise=true , address=145.79.227.29:4967   , piece_id=6GLQROE5CKKU4HYBLDXULBKY7GSQR3K3BNLUGNP3Q244T226U46A, seg_pos={0 0}, upload=514659]: Done uploading piece. err: <nil>
Dec 30 02:33:45.235193228 (587h4m45.106276263s) [seg_pos={0 0}, upload=514659]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:33:45.235447733 (587h4m45.106530788s) [noise=true , address=77.90.31.11:4582     , piece_id=53G52XQXHBAAAD6FEZUVOAKRMQADO7ODQB2GNJFOHHYM4ALJUBXA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx): protocol: write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection; write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44280->77.90.31.11:4582: use of closed network connection
Dec 30 02:33:45.235460017 (587h4m45.106543052s) [noise=true , address=77.90.31.11:4582     , piece_id=53G52XQXHBAAAD6FEZUVOAKRMQADO7ODQB2GNJFOHHYM4ALJUBXA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235465226 (587h4m45.106548261s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG): protocol: write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection; write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33164->145.79.227.7:4301: use of closed network connection
Dec 30 02:33:45.235473973 (587h4m45.106557008s) [noise=true , address=145.79.227.7:4301    , piece_id=CNQFLO2GGX3AV46DWDA3YOMW2CGRZEFBPPURV2NUVDSVQZAE7VOA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235655508 (587h4m45.106738543s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: expected piece hash; context canceled; EOF
Dec 30 02:33:45.235664416 (587h4m45.106747471s) [noise=true , address=145.79.227.33:4894   , piece_id=E5L6XJKWBHQ2SAYQJTSXAHHAW4V6OFNPMIJ73UUYIIHGK5KVLCYA, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235778813 (587h4m45.106861859s) [noise=true , address=45.91.116.9:4280     , piece_id=GU64PSW7MAS6EJQ7ELRDUXDUXCPB5OYX7Q2NELUK5IM33QQAHTDQ, seg_pos={0 0}, upload=514659]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: read tcp 188.166.225.170:37200->45.91.116.9:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 02:33:45.235785917 (587h4m45.106868962s) [noise=true , address=45.91.116.9:4280     , piece_id=GU64PSW7MAS6EJQ7ELRDUXDUXCPB5OYX7Q2NELUK5IM33QQAHTDQ, seg_pos={0 0}, upload=514659]: Long tail cancellation due to optimal threshold
Dec 30 02:33:45.235800013 (587h4m45.106883058s) [seg_pos={0 0}, upload=514659]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:33:45.235869375 (587h4m45.10695241s) [upload=514659]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:33:45.235873674 (587h4m45.106956709s) [upload=514659]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:33:45.520374428 (587h4m45.391457473s) [upload=514659]: Done uploading segments...
Dec 30 02:33:45.520402863 (587h4m45.391485908s) [upload=514659]: Upload finished. err: <nil>
Duration: 45.884603861s core uplink us-select-1 us1-placement-20