Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 354153 reports (Page 210 of 7084)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...]
duration: 
5.867540085s
trace: 
https://jaeger.dev.storj.io/trace/1acb904f4327d6eb
FAIL (2025-12-30 09:16:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded
======LOGS======
6m22.461911784s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment...
Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment.
Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment
Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments...
Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection
Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection
Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection
Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments...
Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil>
Duration: 5.86755898s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
11 out of 60 is failed (threshold=6) [......X....X.....X.........X...X.X....X.X...X........X.....X]
duration: 
5.000815056s
trace: 
https://jaeger.dev.storj.io/trace/731d29a169fda304
FAIL (2025-12-30 09:16:39 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000836197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [X...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X..]
duration: 
5.969337276s
trace: 
https://jaeger.dev.storj.io/trace/1acb904f4327d6eb
FAIL (2025-12-30 09:16:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded
======LOGS======
6m22.461911784s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment...
Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment.
Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment
Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments...
Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection
Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection
Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection
Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments...
Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil>
Duration: 5.969364558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [XX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X.]
duration: 
6.119780896s
trace: 
https://jaeger.dev.storj.io/trace/1acb904f4327d6eb
FAIL (2025-12-30 09:15:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded
======LOGS======
6m22.461911784s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment...
Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment.
Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment
Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments...
Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection
Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection
Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection
Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments...
Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil>
Duration: 6.119795533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [........X.....................X..........................X..]
duration: 
15.264178044s
trace: 
https://jaeger.dev.storj.io/trace/18468de8cb6443cc
FAIL (2025-12-30 09:15:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
oise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment
Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments...
Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174   , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714     , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces.
Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection
Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments...
Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil>
Duration: 15.264195738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X]
duration: 
30.00218075s
trace: 
https://jaeger.dev.storj.io/trace/1acb904f4327d6eb
FAIL (2025-12-30 09:15:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded
======LOGS======
6m22.461911784s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment...
Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment.
Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece...
Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment
Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments...
Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085   , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943   , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473   , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149    , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482   , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330    , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978   , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619   , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911   , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230   , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842     , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299     , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118    , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil>
Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection
Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192    , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection
Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410   , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection
Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122   , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold
Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments...
Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil>
Duration: 30.002206951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [.......X....X.....X.........X...X.X....X.X...X........X.....]
duration: 
5.000705993s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:15:38 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000731912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [........X....X.....X.........X...X.X....X.X...X........X....]
duration: 
5.000957408s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:15:02 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000983898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.........X.....................X..........................X.]
duration: 
21.518248392s
trace: 
https://jaeger.dev.storj.io/trace/18468de8cb6443cc
FAIL (2025-12-30 09:14:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
oise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment
Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments...
Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174   , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714     , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces.
Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection
Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments...
Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil>
Duration: 21.51826399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......]
duration: 
5.716469634s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:14:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 5.716497617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..........X.....................X..........................X]
duration: 
59.244007161s
trace: 
https://jaeger.dev.storj.io/trace/18468de8cb6443cc
FAIL (2025-12-30 09:14:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
oise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece...
Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment
Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments...
Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174   , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495   , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758   , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106   , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347   , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967   , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516   , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060   , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862     , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779     , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714     , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203   , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785   , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil>
Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces.
Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection
Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710    , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215    , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886   , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF
Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649     , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold
Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments...
Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil>
Duration: 59.24403311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [.........X....X.....X.........X...X.X....X.X...X........X...]
duration: 
3.66347834s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:14:00 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.663498286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX.....]
duration: 
6.182399584s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:13:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 6.182413851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX....]
duration: 
6.651237185s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:13:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 6.651264749s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [..........X....X.....X.........X...X.X....X.X...X........X..]
duration: 
3.546461399s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:13:08 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.546489333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X.....................X..........................]
duration: 
29.469710686s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:13:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 29.46973963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX...]
duration: 
5.825308973s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:13:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 5.82533903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX..]
duration: 
6.144246651s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:12:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 6.144274103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............X.....................X.........................]
duration: 
38.66283598s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:12:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 38.662866828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [...........X....X.....X.........X...X.X....X.X...X........X.]
duration: 
3.552764963s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:12:03 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.552793148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX.]
duration: 
5.611117266s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:11:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 5.611131464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [............X....X.....X.........X...X.X....X.X...X........X]
duration: 
5.001180518s
trace: 
https://jaeger.dev.storj.io/trace/30e1221c2da57697
FAIL (2025-12-30 09:11:14 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001206657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X.....................X........................]
duration: 
38.287689605s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:11:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 38.287721146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX]
duration: 
30.003351355s
trace: 
https://jaeger.dev.storj.io/trace/62ea86697d2d9b63
FAIL (2025-12-30 09:10:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload
Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload
Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments...
Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false
Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment...
Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment.
Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece...
Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment
Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments...
Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257    , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229   , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876   , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834   , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186    , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411   , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729   , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518   , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770     , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736     , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974     , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261     , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661     , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil>
Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482   , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled
Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906   , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold
Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments...
Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil>
Duration: 30.00337504s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.............X....X.....X.........X...X.X....X.X...X........]
duration: 
4.676920189s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:10:22 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.676955156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..X]
duration: 
30.002598274s
trace: 
https://jaeger.dev.storj.io/trace/b1a0e5e34a7132a
FAIL (2025-12-30 09:10:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
30 09:09:39.130855887 (593h40m39.001938922s) [upload=520453]: Starting upload
Dec 30 09:09:39.130919378 (593h40m39.002002424s) [upload=520453]: Done starting upload
Dec 30 09:09:39.130977418 (593h40m39.002060463s) [upload=520453]: Uploading segments...
Dec 30 09:09:39.131587821 (593h40m39.002670856s) [upload=520453]: Got next segment. Inline: false
Dec 30 09:09:39.131669266 (593h40m39.002752311s) [upload=520453]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:09:39.131676210 (593h40m39.002759255s) [upload=520453]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:09:39.376261380 (593h40m39.247344415s) [seg_pos={0 0}, upload=520453]: Begin upload segment...
Dec 30 09:09:39.376312157 (593h40m39.247395192s) [seg_pos={0 0}, upload=520453]: Done begin upload segment.
Dec 30 09:09:39.376352243 (593h40m39.247435289s) [noise=true , address=145.79.227.19:4077   , piece_id=ERDOY76J6FC3DM34OULMJJ2BPKGPY5SBZ36QVLK6AREQEVLY6GNQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.376382441 (593h40m39.247465496s) [noise=true , address=145.79.227.52:4446   , piece_id=K332ZXG6ENQEBNHEQHW3HRN4CCU62TCV2C3VCDI7GDXKB6O2EXXA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.376967023 (593h40m39.248050079s) [noise=true , address=45.91.116.14:4190    , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.377381593 (593h40m39.248464638s) [noise=true , address=77.90.31.17:4770     , piece_id=QXQX4JTHL52H42PED4WRIZEOHUM2X3G47ZAQOIT3ON7THXGLULJQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.377778890 (593h40m39.248861935s) [noise=true , address=77.90.31.14:4663     , piece_id=ADSB7SA2NOZJT355GF4IC2XNKLUH5ULBY7YASKC3TGRELMIFPE6Q, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.378213938 (593h40m39.249296973s) [noise=true , address=145.79.227.46:4603   , piece_id=6Q7TLTEPRTIXRBGBZVT6P5TDXDALOOBSKHPKBT7WECR7IWOZNHDA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.378607258 (593h40m39.249690303s) [noise=true , address=77.90.31.18:4805     , piece_id=TXATSZUISJDD4BJ3NBVBWRUFHEJZ4IWLPLAD62TGMGAHNMMSA3UA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.379084325 (593h40m39.25016736s) [noise=true , address=145.79.227.13:4803   , piece_id=7HJMOGDMPPWCZ2LV74PSF56XQWJFWRJVSSQYP3XDNOGV4EPVOAKQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.379759171 (593h40m39.250842216s) [noise=true , address=145.79.227.16:4911   , piece_id=N6GOA4IXVQGR2EP2MNOCCKZS3UZYTG7HUHCHCZJUP7G3IH4QVRGQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.380273610 (593h40m39.251356645s) [noise=true , address=145.79.227.4:4186    , piece_id=YIA7NNRLQXCTZH7HZGV2YP6BNBF7E3AD37FE63K2Y3WMSQTT6ISA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.380709821 (593h40m39.251792866s) [noise=true , address=77.90.31.16:4736     , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.381215865 (593h40m39.25229891s) [noise=true , address=145.79.227.49:4339   , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.381666702 (593h40m39.252749748s) [noise=true , address=45.91.116.7:4012     , piece_id=QSMNKZWDOJEE73KUN6ES4ZYPYSXT2KA5RBLSOSLQLWIN2RODO7IA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.382104286 (593h40m39.253187331s) [noise=true , address=145.79.227.27:4691   , piece_id=ZKQ6HAZQ7Z4WMGBHMUABHPOYMVTHYTXSE74U4QUJWPYWM6Z6XZBA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.382660405 (593h40m39.25374345s) [noise=true , address=145.79.227.6:4257    , piece_id=ZYQSH3EVJDECI4HIZMORI55AAAUU23DJFNTOLJTOQR72WEOPF7BA, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:39.382988189 (593h40m39.254071224s) [noise=true , address=145.79.227.11:4732   , piece_id=MP35K3UPLRZSTIOKVBPHX5TABOAFSXE3IO5SPG33SB2KVG2LWTFQ, seg_pos={0 0}, upload=520453]: Uploading piece...
Dec 30 09:09:41.087167548 (593h40m40.958250593s) [upload=520453]: Next returned nil segment
Dec 30 09:09:41.087194289 (593h40m40.958277334s) [upload=520453]: Waiting for error group managing segments...
Dec 30 09:09:41.368994144 (593h40m41.240077189s) [noise=true , address=145.79.227.13:4803   , piece_id=7HJMOGDMPPWCZ2LV74PSF56XQWJFWRJVSSQYP3XDNOGV4EPVOAKQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.370028152 (593h40m41.241111197s) [noise=true , address=145.79.227.52:4446   , piece_id=K332ZXG6ENQEBNHEQHW3HRN4CCU62TCV2C3VCDI7GDXKB6O2EXXA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.377031237 (593h40m41.248114272s) [noise=true , address=145.79.227.19:4077   , piece_id=ERDOY76J6FC3DM34OULMJJ2BPKGPY5SBZ36QVLK6AREQEVLY6GNQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.378687892 (593h40m41.249770927s) [noise=true , address=145.79.227.16:4911   , piece_id=N6GOA4IXVQGR2EP2MNOCCKZS3UZYTG7HUHCHCZJUP7G3IH4QVRGQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.382652120 (593h40m41.253735165s) [noise=true , address=145.79.227.46:4603   , piece_id=6Q7TLTEPRTIXRBGBZVT6P5TDXDALOOBSKHPKBT7WECR7IWOZNHDA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.385120671 (593h40m41.256203736s) [noise=true , address=145.79.227.4:4186    , piece_id=YIA7NNRLQXCTZH7HZGV2YP6BNBF7E3AD37FE63K2Y3WMSQTT6ISA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.388830875 (593h40m41.25991391s) [noise=true , address=145.79.227.6:4257    , piece_id=ZYQSH3EVJDECI4HIZMORI55AAAUU23DJFNTOLJTOQR72WEOPF7BA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.392115169 (593h40m41.263198204s) [noise=true , address=145.79.227.11:4732   , piece_id=MP35K3UPLRZSTIOKVBPHX5TABOAFSXE3IO5SPG33SB2KVG2LWTFQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.441332297 (593h40m41.312415332s) [noise=true , address=145.79.227.27:4691   , piece_id=ZKQ6HAZQ7Z4WMGBHMUABHPOYMVTHYTXSE74U4QUJWPYWM6Z6XZBA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.694832283 (593h40m41.565915328s) [noise=true , address=45.91.116.7:4012     , piece_id=QSMNKZWDOJEE73KUN6ES4ZYPYSXT2KA5RBLSOSLQLWIN2RODO7IA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.711756859 (593h40m41.582839895s) [noise=true , address=77.90.31.14:4663     , piece_id=ADSB7SA2NOZJT355GF4IC2XNKLUH5ULBY7YASKC3TGRELMIFPE6Q, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:41.977416800 (593h40m41.848499835s) [noise=true , address=77.90.31.17:4770     , piece_id=QXQX4JTHL52H42PED4WRIZEOHUM2X3G47ZAQOIT3ON7THXGLULJQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:42.218243134 (593h40m42.089326169s) [noise=true , address=77.90.31.18:4805     , piece_id=TXATSZUISJDD4BJ3NBVBWRUFHEJZ4IWLPLAD62TGMGAHNMMSA3UA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil>
Dec 30 09:09:42.218260999 (593h40m42.089344034s) [seg_pos={0 0}, upload=520453]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:09:42.218450148 (593h40m42.089533193s) [noise=true , address=145.79.227.49:4339   , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 09:09:42.218456089 (593h40m42.089539124s) [noise=true , address=145.79.227.49:4339   , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold
Dec 30 09:09:42.218676951 (593h40m42.089759986s) [noise=true , address=45.91.116.14:4190    , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection
Dec 30 09:09:42.218684174 (593h40m42.089767209s) [noise=true , address=45.91.116.14:4190    , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold
Dec 30 09:09:42.218815815 (593h40m42.08989886s) [noise=true , address=77.90.31.16:4736     , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 30 09:09:42.218822007 (593h40m42.089905042s) [noise=true , address=77.90.31.16:4736     , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold
Dec 30 09:09:42.218836123 (593h40m42.089919158s) [seg_pos={0 0}, upload=520453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:09:42.218912538 (593h40m42.089995574s) [upload=520453]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:09:42.218932998 (593h40m42.090016033s) [upload=520453]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:09:42.508927415 (593h40m42.38001045s) [upload=520453]: Done uploading segments...
Dec 30 09:09:42.508946511 (593h40m42.380029556s) [upload=520453]: Upload finished. err: <nil>
Duration: 30.00262187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X.....................X.......................]
duration: 
15.154987182s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:09:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 15.155016067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [..............X....X.....X.........X...X.X....X.X...X.......]
duration: 
5.001127755s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:09:30 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001151349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [...............X....X.....X.........X...X.X....X.X...X......]
duration: 
3.565165847s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:09:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.565322465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X.....................X......................]
duration: 
38.874158444s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:09:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 38.874189311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..]
duration: 
6.007206706s
trace: 
https://jaeger.dev.storj.io/trace/5c53eea50bba27b
FAIL (2025-12-30 09:09:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ) context deadline exceeded
======LOGS======
:08:10.690779532 (593h39m10.561862577s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:08:10.690786285 (593h39m10.56186932s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:08:10.935761318 (593h39m10.806844353s) [seg_pos={0 0}, upload=520429]: Begin upload segment...
Dec 30 09:08:10.935816012 (593h39m10.806899047s) [seg_pos={0 0}, upload=520429]: Done begin upload segment.
Dec 30 09:08:10.935848785 (593h39m10.80693182s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.935945799 (593h39m10.807028865s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936630543 (593h39m10.807713588s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936760530 (593h39m10.807843566s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937005657 (593h39m10.808088692s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937221167 (593h39m10.808304202s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937276663 (593h39m10.808359699s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937529004 (593h39m10.808612039s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937674771 (593h39m10.808757816s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937824377 (593h39m10.808907412s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938148073 (593h39m10.809231108s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938240058 (593h39m10.809323103s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938599753 (593h39m10.809682788s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938780837 (593h39m10.809863883s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938888933 (593h39m10.809971978s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.939302892 (593h39m10.810385937s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:12.644838984 (593h39m12.515922019s) [upload=520429]: Next returned nil segment
Dec 30 09:08:12.644852830 (593h39m12.515935865s) [upload=520429]: Waiting for error group managing segments...
Dec 30 09:08:12.929694426 (593h39m12.800777471s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.933169093 (593h39m12.804252128s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.944748601 (593h39m12.815831636s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.951606620 (593h39m12.822689655s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.011305357 (593h39m12.882388402s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.311682716 (593h39m13.182765751s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.878441738 (593h39m13.749524783s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.931694179 (593h39m13.802777225s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.956336944 (593h39m13.827419989s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:16.396503654 (593h39m16.267586699s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:17.359300108 (593h39m17.230383143s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:18.044962994 (593h39m17.916046029s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793143977 (593h39m19.664227032s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793166420 (593h39m19.664249466s) [seg_pos={0 0}, upload=520429]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:08:19.793446243 (593h39m19.664529288s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection
Dec 30 09:08:19.793453858 (593h39m19.664536903s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793651063 (593h39m19.664734098s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: read tcp 188.166.225.170:42626->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 09:08:19.793666532 (593h39m19.664749567s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793697882 (593h39m19.664780917s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection
Dec 30 09:08:19.793706007 (593h39m19.664789042s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793725735 (593h39m19.66480877s) [seg_pos={0 0}, upload=520429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:08:19.793825996 (593h39m19.664909031s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:08:19.793830825 (593h39m19.664913861s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:08:20.078348705 (593h39m19.94943174s) [upload=520429]: Done uploading segments...
Dec 30 09:08:20.078371338 (593h39m19.949454383s) [upload=520429]: Upload finished. err: <nil>
Duration: 6.007235981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX.]
duration: 
5.851487633s
trace: 
https://jaeger.dev.storj.io/trace/5c53eea50bba27b
FAIL (2025-12-30 09:08:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ) context deadline exceeded
======LOGS======
:08:10.690779532 (593h39m10.561862577s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:08:10.690786285 (593h39m10.56186932s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:08:10.935761318 (593h39m10.806844353s) [seg_pos={0 0}, upload=520429]: Begin upload segment...
Dec 30 09:08:10.935816012 (593h39m10.806899047s) [seg_pos={0 0}, upload=520429]: Done begin upload segment.
Dec 30 09:08:10.935848785 (593h39m10.80693182s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.935945799 (593h39m10.807028865s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936630543 (593h39m10.807713588s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936760530 (593h39m10.807843566s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937005657 (593h39m10.808088692s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937221167 (593h39m10.808304202s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937276663 (593h39m10.808359699s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937529004 (593h39m10.808612039s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937674771 (593h39m10.808757816s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937824377 (593h39m10.808907412s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938148073 (593h39m10.809231108s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938240058 (593h39m10.809323103s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938599753 (593h39m10.809682788s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938780837 (593h39m10.809863883s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938888933 (593h39m10.809971978s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.939302892 (593h39m10.810385937s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:12.644838984 (593h39m12.515922019s) [upload=520429]: Next returned nil segment
Dec 30 09:08:12.644852830 (593h39m12.515935865s) [upload=520429]: Waiting for error group managing segments...
Dec 30 09:08:12.929694426 (593h39m12.800777471s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.933169093 (593h39m12.804252128s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.944748601 (593h39m12.815831636s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.951606620 (593h39m12.822689655s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.011305357 (593h39m12.882388402s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.311682716 (593h39m13.182765751s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.878441738 (593h39m13.749524783s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.931694179 (593h39m13.802777225s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.956336944 (593h39m13.827419989s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:16.396503654 (593h39m16.267586699s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:17.359300108 (593h39m17.230383143s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:18.044962994 (593h39m17.916046029s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793143977 (593h39m19.664227032s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793166420 (593h39m19.664249466s) [seg_pos={0 0}, upload=520429]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:08:19.793446243 (593h39m19.664529288s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection
Dec 30 09:08:19.793453858 (593h39m19.664536903s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793651063 (593h39m19.664734098s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: read tcp 188.166.225.170:42626->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 09:08:19.793666532 (593h39m19.664749567s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793697882 (593h39m19.664780917s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection
Dec 30 09:08:19.793706007 (593h39m19.664789042s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793725735 (593h39m19.66480877s) [seg_pos={0 0}, upload=520429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:08:19.793825996 (593h39m19.664909031s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:08:19.793830825 (593h39m19.664913861s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:08:20.078348705 (593h39m19.94943174s) [upload=520429]: Done uploading segments...
Dec 30 09:08:20.078371338 (593h39m19.949454383s) [upload=520429]: Upload finished. err: <nil>
Duration: 5.851502842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX]
duration: 
30.0029919s
trace: 
https://jaeger.dev.storj.io/trace/5c53eea50bba27b
FAIL (2025-12-30 09:08:40 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ) context deadline exceeded
======LOGS======
:08:10.690779532 (593h39m10.561862577s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:08:10.690786285 (593h39m10.56186932s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:08:10.935761318 (593h39m10.806844353s) [seg_pos={0 0}, upload=520429]: Begin upload segment...
Dec 30 09:08:10.935816012 (593h39m10.806899047s) [seg_pos={0 0}, upload=520429]: Done begin upload segment.
Dec 30 09:08:10.935848785 (593h39m10.80693182s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.935945799 (593h39m10.807028865s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936630543 (593h39m10.807713588s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.936760530 (593h39m10.807843566s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937005657 (593h39m10.808088692s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937221167 (593h39m10.808304202s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937276663 (593h39m10.808359699s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937529004 (593h39m10.808612039s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937674771 (593h39m10.808757816s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.937824377 (593h39m10.808907412s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938148073 (593h39m10.809231108s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938240058 (593h39m10.809323103s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938599753 (593h39m10.809682788s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938780837 (593h39m10.809863883s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.938888933 (593h39m10.809971978s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:10.939302892 (593h39m10.810385937s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Uploading piece...
Dec 30 09:08:12.644838984 (593h39m12.515922019s) [upload=520429]: Next returned nil segment
Dec 30 09:08:12.644852830 (593h39m12.515935865s) [upload=520429]: Waiting for error group managing segments...
Dec 30 09:08:12.929694426 (593h39m12.800777471s) [noise=true , address=145.79.227.17:4972   , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.933169093 (593h39m12.804252128s) [noise=true , address=145.79.227.37:4567   , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.944748601 (593h39m12.815831636s) [noise=true , address=145.79.227.3:4150    , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:12.951606620 (593h39m12.822689655s) [noise=true , address=145.79.227.20:4439   , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.011305357 (593h39m12.882388402s) [noise=true , address=145.79.227.32:4836   , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.311682716 (593h39m13.182765751s) [noise=true , address=45.91.116.11:4117    , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.878441738 (593h39m13.749524783s) [noise=true , address=77.90.31.21:4877     , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.931694179 (593h39m13.802777225s) [noise=true , address=45.91.116.9:4263     , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:13.956336944 (593h39m13.827419989s) [noise=true , address=77.90.31.15:4697     , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:16.396503654 (593h39m16.267586699s) [noise=true , address=145.79.227.5:4221    , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:17.359300108 (593h39m17.230383143s) [noise=true , address=145.79.227.14:4839   , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:18.044962994 (593h39m17.916046029s) [noise=true , address=145.79.227.36:4517   , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793143977 (593h39m19.664227032s) [noise=true , address=145.79.227.16:4909   , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil>
Dec 30 09:08:19.793166420 (593h39m19.664249466s) [seg_pos={0 0}, upload=520429]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:08:19.793446243 (593h39m19.664529288s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection
Dec 30 09:08:19.793453858 (593h39m19.664536903s) [noise=true , address=145.79.227.50:4373   , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793651063 (593h39m19.664734098s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: read tcp 188.166.225.170:42626->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 09:08:19.793666532 (593h39m19.664749567s) [noise=true , address=145.79.227.45:4231   , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793697882 (593h39m19.664780917s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection
Dec 30 09:08:19.793706007 (593h39m19.664789042s) [noise=true , address=145.79.227.46:4601   , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold
Dec 30 09:08:19.793725735 (593h39m19.66480877s) [seg_pos={0 0}, upload=520429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:08:19.793825996 (593h39m19.664909031s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:08:19.793830825 (593h39m19.664913861s) [upload=520429]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:08:20.078348705 (593h39m19.94943174s) [upload=520429]: Done uploading segments...
Dec 30 09:08:20.078371338 (593h39m19.949454383s) [upload=520429]: Upload finished. err: <nil>
Duration: 30.003006567s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [................X....X.....X.........X...X.X....X.X...X.....]
duration: 
3.938481389s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:08:15 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.938507811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................X.....................X.....................]
duration: 
46.182461367s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:08:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 46.182487606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..X]
duration: 
30.002949279s
trace: 
https://jaeger.dev.storj.io/trace/1219e302ea814c6a
FAIL (2025-12-30 09:08:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 6OLBRWPRVC3O5CZ4CIPQPQQNMTDKVFAWDL6W5FDVMLVRQTBZFO7Q) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: G5AHGDU66OCXHH7NCB6UJ32D4EQRSEJBHKYSOM7KWRXMHBV6SMRQ) context deadline exceeded
======LOGS======
Dec 30 09:07:40.466645187 (593h38m40.337728232s) [upload=520419]: Starting upload
Dec 30 09:07:40.466705713 (593h38m40.337788758s) [upload=520419]: Done starting upload
Dec 30 09:07:40.466864294 (593h38m40.337947329s) [upload=520419]: Uploading segments...
Dec 30 09:07:40.466936181 (593h38m40.338019216s) [upload=520419]: Got next segment. Inline: false
Dec 30 09:07:40.466981828 (593h38m40.338064873s) [upload=520419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:07:40.466990074 (593h38m40.338073119s) [upload=520419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:07:40.713944548 (593h38m40.585027683s) [seg_pos={0 0}, upload=520419]: Begin upload segment...
Dec 30 09:07:40.713998901 (593h38m40.585081936s) [seg_pos={0 0}, upload=520419]: Done begin upload segment.
Dec 30 09:07:40.714031553 (593h38m40.585114598s) [noise=true , address=145.79.227.43:4159   , piece_id=OBLKHYEGYUCFMWS63NM7MEAM3LV3ZPCFONN7TRG43EBS4AYOUJBQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.714105605 (593h38m40.58518865s) [noise=true , address=77.90.31.19:4841     , piece_id=G5AHGDU66OCXHH7NCB6UJ32D4EQRSEJBHKYSOM7KWRXMHBV6SMRQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.715918938 (593h38m40.587001973s) [noise=true , address=145.79.227.5:4221    , piece_id=GI7TAO4SUOGRT3V3EOHXN6SBVK2ADDNZGMJSWH6W5WBCPIOC4QGQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.715933014 (593h38m40.587016059s) [noise=true , address=45.91.116.7:4012     , piece_id=6Z2OI6YKZNYCEQ4ZBGH74C2ZYFUGC6D6GFX74NJMLFSFQBXMIVHQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.716283301 (593h38m40.587366336s) [noise=true , address=145.79.227.17:4971   , piece_id=HGZH6YBIKIISSGBVH6MIA26F46PDGV2NPAMAGDZ7DJAPGVDPYSIA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.716367372 (593h38m40.587450417s) [noise=true , address=45.91.116.5:4299     , piece_id=JMTRVXMUFGVMYA6FMXY3KFHODMJYGG3UWUN4IBCPBAJF4CECMG3A, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.716666501 (593h38m40.587749536s) [noise=true , address=145.79.227.41:4087   , piece_id=OWD5XE2OPESZXSWSZRA3SCAFJ427EC325PWQOY33MHJYYQICFTCQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.716776570 (593h38m40.587859615s) [noise=true , address=145.79.227.22:4510   , piece_id=BM6RWRPDWYNV5OZO2CXUSE7K2NKUU26725TWEFE4HZNXUQVS75FA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.717017138 (593h38m40.588100173s) [noise=true , address=77.90.31.10:4519     , piece_id=RNQSG4LXQUMUQ3KAPYQFLY5B27LLE4IMLFMMTUQOVXTGKNJJ3KOA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.717187232 (593h38m40.588270267s) [noise=true , address=45.91.116.9:4264     , piece_id=6OLBRWPRVC3O5CZ4CIPQPQQNMTDKVFAWDL6W5FDVMLVRQTBZFO7Q, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.717413353 (593h38m40.588496388s) [noise=true , address=145.79.227.14:4839   , piece_id=BYWYJZY74HZMTTE26NCHMYRFMF6TB6K524IOAR2YXETP6WGYRHDA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.717623684 (593h38m40.588706729s) [noise=true , address=145.79.227.42:4121   , piece_id=ISXWPM3L3OIHUN6JEV6K2ORKHJ6F2BCTLEL4WX4JSLY2DSJZAM3A, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.717755173 (593h38m40.588838218s) [noise=true , address=145.79.227.48:4301   , piece_id=WYXAYJGWRDLC7BJOALOYJLIY225I7M3EUGRKKUZE3OYRVU532XAA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.718132151 (593h38m40.589215186s) [noise=true , address=145.79.227.11:4729   , piece_id=V5L2EGYYQHZLV76BEXMTXOTWX4UXA3QPD6IKMYC7O5VBNGNVHDKQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.718170514 (593h38m40.589253549s) [noise=true , address=77.90.31.18:4805     , piece_id=XV6OUT22NYK227DFI5JFQZXBZ4TDDHXTGBKK4JKD4HPSWDCW2WZA, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:40.719062383 (593h38m40.590145418s) [noise=true , address=145.79.227.4:4187    , piece_id=VJH2ZWMGMY4RSTNPB4WTHKVNHSJFVNUMXAI5BTFSB52S7PZPXBGQ, seg_pos={0 0}, upload=520419]: Uploading piece...
Dec 30 09:07:42.321440335 (593h38m42.19252338s) [upload=520419]: Next returned nil segment
Dec 30 09:07:42.321457037 (593h38m42.192540103s) [upload=520419]: Waiting for error group managing segments...
Dec 30 09:07:42.692006325 (593h38m42.56308936s) [noise=true , address=145.79.227.17:4971   , piece_id=HGZH6YBIKIISSGBVH6MIA26F46PDGV2NPAMAGDZ7DJAPGVDPYSIA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.704356832 (593h38m42.575439877s) [noise=true , address=145.79.227.48:4301   , piece_id=WYXAYJGWRDLC7BJOALOYJLIY225I7M3EUGRKKUZE3OYRVU532XAA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.704677493 (593h38m42.575760528s) [noise=true , address=145.79.227.11:4729   , piece_id=V5L2EGYYQHZLV76BEXMTXOTWX4UXA3QPD6IKMYC7O5VBNGNVHDKQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.706396706 (593h38m42.577479751s) [noise=true , address=145.79.227.14:4839   , piece_id=BYWYJZY74HZMTTE26NCHMYRFMF6TB6K524IOAR2YXETP6WGYRHDA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.720816745 (593h38m42.59189979s) [noise=true , address=145.79.227.43:4159   , piece_id=OBLKHYEGYUCFMWS63NM7MEAM3LV3ZPCFONN7TRG43EBS4AYOUJBQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.724711440 (593h38m42.595794485s) [noise=true , address=145.79.227.5:4221    , piece_id=GI7TAO4SUOGRT3V3EOHXN6SBVK2ADDNZGMJSWH6W5WBCPIOC4QGQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.728808582 (593h38m42.599891627s) [noise=true , address=145.79.227.4:4187    , piece_id=VJH2ZWMGMY4RSTNPB4WTHKVNHSJFVNUMXAI5BTFSB52S7PZPXBGQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.728885579 (593h38m42.599968614s) [noise=true , address=145.79.227.22:4510   , piece_id=BM6RWRPDWYNV5OZO2CXUSE7K2NKUU26725TWEFE4HZNXUQVS75FA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.740112216 (593h38m42.611195251s) [noise=true , address=145.79.227.42:4121   , piece_id=ISXWPM3L3OIHUN6JEV6K2ORKHJ6F2BCTLEL4WX4JSLY2DSJZAM3A, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.810119091 (593h38m42.681202126s) [noise=true , address=145.79.227.41:4087   , piece_id=OWD5XE2OPESZXSWSZRA3SCAFJ427EC325PWQOY33MHJYYQICFTCQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.892519255 (593h38m42.76360229s) [noise=true , address=77.90.31.19:4841     , piece_id=G5AHGDU66OCXHH7NCB6UJ32D4EQRSEJBHKYSOM7KWRXMHBV6SMRQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:42.927854138 (593h38m42.798937183s) [noise=true , address=77.90.31.10:4519     , piece_id=RNQSG4LXQUMUQ3KAPYQFLY5B27LLE4IMLFMMTUQOVXTGKNJJ3KOA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:43.018965960 (593h38m42.890049005s) [noise=true , address=45.91.116.9:4264     , piece_id=6OLBRWPRVC3O5CZ4CIPQPQQNMTDKVFAWDL6W5FDVMLVRQTBZFO7Q, seg_pos={0 0}, upload=520419]: Done uploading piece. err: <nil>
Dec 30 09:07:43.018991238 (593h38m42.890074273s) [seg_pos={0 0}, upload=520419]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:07:43.019229452 (593h38m42.890312487s) [noise=true , address=45.91.116.7:4012     , piece_id=6Z2OI6YKZNYCEQ4ZBGH74C2ZYFUGC6D6GFX74NJMLFSFQBXMIVHQ, seg_pos={0 0}, upload=520419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:56348->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:56348->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56348->45.91.116.7:4012: use of closed network connection
Dec 30 09:07:43.019268045 (593h38m42.89035108s) [noise=true , address=45.91.116.7:4012     , piece_id=6Z2OI6YKZNYCEQ4ZBGH74C2ZYFUGC6D6GFX74NJMLFSFQBXMIVHQ, seg_pos={0 0}, upload=520419]: Long tail cancellation due to optimal threshold
Dec 30 09:07:43.019283294 (593h38m42.890366339s) [noise=true , address=45.91.116.5:4299     , piece_id=JMTRVXMUFGVMYA6FMXY3KFHODMJYGG3UWUN4IBCPBAJF4CECMG3A, seg_pos={0 0}, upload=520419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 30 09:07:43.019292392 (593h38m42.890375437s) [noise=true , address=45.91.116.5:4299     , piece_id=JMTRVXMUFGVMYA6FMXY3KFHODMJYGG3UWUN4IBCPBAJF4CECMG3A, seg_pos={0 0}, upload=520419]: Long tail cancellation due to optimal threshold
Dec 30 09:07:43.019517149 (593h38m42.890600204s) [noise=true , address=77.90.31.18:4805     , piece_id=XV6OUT22NYK227DFI5JFQZXBZ4TDDHXTGBKK4JKD4HPSWDCW2WZA, seg_pos={0 0}, upload=520419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 30 09:07:43.019526437 (593h38m42.890609492s) [noise=true , address=77.90.31.18:4805     , piece_id=XV6OUT22NYK227DFI5JFQZXBZ4TDDHXTGBKK4JKD4HPSWDCW2WZA, seg_pos={0 0}, upload=520419]: Long tail cancellation due to optimal threshold
Dec 30 09:07:43.019546305 (593h38m42.89062935s) [seg_pos={0 0}, upload=520419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:07:43.019735365 (593h38m42.89081841s) [upload=520419]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:07:43.019742288 (593h38m42.890825333s) [upload=520419]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:07:43.307774600 (593h38m43.178857635s) [upload=520419]: Done uploading segments...
Dec 30 09:07:43.307798055 (593h38m43.17888109s) [upload=520419]: Upload finished. err: <nil>
Duration: 30.002974608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.................X....X.....X.........X...X.X....X.X...X....]
duration: 
3.697504633s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:07:27 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.697531374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.................X.....................X....................]
duration: 
39.090004314s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:07:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 39.090031597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..]
duration: 
6.045780307s
trace: 
https://jaeger.dev.storj.io/trace/7ac28e43b7dc3749
FAIL (2025-12-30 09:06:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ) context deadline exceeded
======LOGS======
 09:05:30.498020774 (593h36m30.369103809s) [upload=520388]: Starting upload
Dec 30 09:05:30.498099945 (593h36m30.36918298s) [upload=520388]: Done starting upload
Dec 30 09:05:30.498233549 (593h36m30.369316594s) [upload=520388]: Uploading segments...
Dec 30 09:05:30.498341164 (593h36m30.369424209s) [upload=520388]: Got next segment. Inline: false
Dec 30 09:05:30.498427678 (593h36m30.369510724s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:05:30.498435032 (593h36m30.369518077s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:05:30.743518531 (593h36m30.614601576s) [seg_pos={0 0}, upload=520388]: Begin upload segment...
Dec 30 09:05:30.743677635 (593h36m30.61476068s) [seg_pos={0 0}, upload=520388]: Done begin upload segment.
Dec 30 09:05:30.743745563 (593h36m30.614828598s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.743929113 (593h36m30.615012148s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.744451087 (593h36m30.615534122s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745053314 (593h36m30.616136359s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745677522 (593h36m30.616760557s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.746052717 (593h36m30.617135762s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748134841 (593h36m30.619217876s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748524993 (593h36m30.619608029s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748852418 (593h36m30.619935453s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.749773531 (593h36m30.620856566s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.750861282 (593h36m30.621944317s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.751413794 (593h36m30.622496829s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.752675798 (593h36m30.623758843s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.756080310 (593h36m30.627163345s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.758102280 (593h36m30.629185325s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.759980777 (593h36m30.631063812s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:32.350420280 (593h36m32.221503325s) [upload=520388]: Next returned nil segment
Dec 30 09:05:32.350436361 (593h36m32.221519406s) [upload=520388]: Waiting for error group managing segments...
Dec 30 09:05:32.733313892 (593h36m32.604396927s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735361480 (593h36m32.606444515s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735965120 (593h36m32.607048155s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.749422064 (593h36m32.620505109s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.752949580 (593h36m32.624032615s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.756720481 (593h36m32.627803516s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.771933810 (593h36m32.643016855s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.774008941 (593h36m32.645091986s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.824307567 (593h36m32.695390602s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.915758335 (593h36m32.78684137s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.930883967 (593h36m32.801967012s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.053179076 (593h36m32.924262112s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095502747 (593h36m32.966585782s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095525922 (593h36m32.966608957s) [seg_pos={0 0}, upload=520388]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:05:33.095818288 (593h36m32.966901323s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095826144 (593h36m32.966909179s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.095853635 (593h36m32.96693668s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095862242 (593h36m32.966945278s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096036795 (593h36m32.96711983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection
Dec 30 09:05:33.096043938 (593h36m32.967126983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096069176 (593h36m32.967152221s) [seg_pos={0 0}, upload=520388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:05:33.096170128 (593h36m32.967253173s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:05:33.096176110 (593h36m32.967259145s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:05:33.389556302 (593h36m33.260639347s) [upload=520388]: Done uploading segments...
Dec 30 09:05:33.389580217 (593h36m33.260663262s) [upload=520388]: Upload finished. err: <nil>
Duration: 6.045806126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X.]
duration: 
5.80845518s
trace: 
https://jaeger.dev.storj.io/trace/7ac28e43b7dc3749
FAIL (2025-12-30 09:06:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ) context deadline exceeded
======LOGS======
 09:05:30.498020774 (593h36m30.369103809s) [upload=520388]: Starting upload
Dec 30 09:05:30.498099945 (593h36m30.36918298s) [upload=520388]: Done starting upload
Dec 30 09:05:30.498233549 (593h36m30.369316594s) [upload=520388]: Uploading segments...
Dec 30 09:05:30.498341164 (593h36m30.369424209s) [upload=520388]: Got next segment. Inline: false
Dec 30 09:05:30.498427678 (593h36m30.369510724s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:05:30.498435032 (593h36m30.369518077s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:05:30.743518531 (593h36m30.614601576s) [seg_pos={0 0}, upload=520388]: Begin upload segment...
Dec 30 09:05:30.743677635 (593h36m30.61476068s) [seg_pos={0 0}, upload=520388]: Done begin upload segment.
Dec 30 09:05:30.743745563 (593h36m30.614828598s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.743929113 (593h36m30.615012148s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.744451087 (593h36m30.615534122s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745053314 (593h36m30.616136359s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745677522 (593h36m30.616760557s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.746052717 (593h36m30.617135762s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748134841 (593h36m30.619217876s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748524993 (593h36m30.619608029s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748852418 (593h36m30.619935453s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.749773531 (593h36m30.620856566s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.750861282 (593h36m30.621944317s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.751413794 (593h36m30.622496829s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.752675798 (593h36m30.623758843s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.756080310 (593h36m30.627163345s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.758102280 (593h36m30.629185325s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.759980777 (593h36m30.631063812s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:32.350420280 (593h36m32.221503325s) [upload=520388]: Next returned nil segment
Dec 30 09:05:32.350436361 (593h36m32.221519406s) [upload=520388]: Waiting for error group managing segments...
Dec 30 09:05:32.733313892 (593h36m32.604396927s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735361480 (593h36m32.606444515s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735965120 (593h36m32.607048155s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.749422064 (593h36m32.620505109s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.752949580 (593h36m32.624032615s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.756720481 (593h36m32.627803516s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.771933810 (593h36m32.643016855s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.774008941 (593h36m32.645091986s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.824307567 (593h36m32.695390602s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.915758335 (593h36m32.78684137s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.930883967 (593h36m32.801967012s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.053179076 (593h36m32.924262112s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095502747 (593h36m32.966585782s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095525922 (593h36m32.966608957s) [seg_pos={0 0}, upload=520388]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:05:33.095818288 (593h36m32.966901323s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095826144 (593h36m32.966909179s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.095853635 (593h36m32.96693668s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095862242 (593h36m32.966945278s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096036795 (593h36m32.96711983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection
Dec 30 09:05:33.096043938 (593h36m32.967126983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096069176 (593h36m32.967152221s) [seg_pos={0 0}, upload=520388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:05:33.096170128 (593h36m32.967253173s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:05:33.096176110 (593h36m32.967259145s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:05:33.389556302 (593h36m33.260639347s) [upload=520388]: Done uploading segments...
Dec 30 09:05:33.389580217 (593h36m33.260663262s) [upload=520388]: Upload finished. err: <nil>
Duration: 5.808485629s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [..................X....X.....X.........X...X.X....X.X...X...]
duration: 
3.81464511s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:06:15 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.814696117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..................X.....................X...................]
duration: 
34.550180161s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:06:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 34.550199327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X]
duration: 
30.003263712s
trace: 
https://jaeger.dev.storj.io/trace/7ac28e43b7dc3749
FAIL (2025-12-30 09:05:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ) context deadline exceeded
======LOGS======
 09:05:30.498020774 (593h36m30.369103809s) [upload=520388]: Starting upload
Dec 30 09:05:30.498099945 (593h36m30.36918298s) [upload=520388]: Done starting upload
Dec 30 09:05:30.498233549 (593h36m30.369316594s) [upload=520388]: Uploading segments...
Dec 30 09:05:30.498341164 (593h36m30.369424209s) [upload=520388]: Got next segment. Inline: false
Dec 30 09:05:30.498427678 (593h36m30.369510724s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:05:30.498435032 (593h36m30.369518077s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:05:30.743518531 (593h36m30.614601576s) [seg_pos={0 0}, upload=520388]: Begin upload segment...
Dec 30 09:05:30.743677635 (593h36m30.61476068s) [seg_pos={0 0}, upload=520388]: Done begin upload segment.
Dec 30 09:05:30.743745563 (593h36m30.614828598s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.743929113 (593h36m30.615012148s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.744451087 (593h36m30.615534122s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745053314 (593h36m30.616136359s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.745677522 (593h36m30.616760557s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.746052717 (593h36m30.617135762s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748134841 (593h36m30.619217876s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748524993 (593h36m30.619608029s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.748852418 (593h36m30.619935453s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.749773531 (593h36m30.620856566s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.750861282 (593h36m30.621944317s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.751413794 (593h36m30.622496829s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.752675798 (593h36m30.623758843s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.756080310 (593h36m30.627163345s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.758102280 (593h36m30.629185325s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:30.759980777 (593h36m30.631063812s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Uploading piece...
Dec 30 09:05:32.350420280 (593h36m32.221503325s) [upload=520388]: Next returned nil segment
Dec 30 09:05:32.350436361 (593h36m32.221519406s) [upload=520388]: Waiting for error group managing segments...
Dec 30 09:05:32.733313892 (593h36m32.604396927s) [noise=true , address=145.79.227.11:4729   , piece_id=ER627QOCWSJSWIWZS3WJFZHNIS7IR7JOP5EZSUSTUYUTH4T2ZEUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735361480 (593h36m32.606444515s) [noise=true , address=145.79.227.6:4259    , piece_id=3YL36RFH2JINLZ536CQ7I2A2SL3PKZGMAE6MCY3VASHGENFRXSQA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.735965120 (593h36m32.607048155s) [noise=true , address=145.79.227.8:4330    , piece_id=OIUHKWMNZ73C7VNWSVU7R7MQFBMIP27DYBABMCXKKLHSE2QZCQSA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.749422064 (593h36m32.620505109s) [noise=true , address=145.79.227.37:4567   , piece_id=KWSD76WCQ2BKVZAWV5MQQWYCNWA3U7QGFEMN5CQWLJMD5ZXGIVSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.752949580 (593h36m32.624032615s) [noise=true , address=145.79.227.36:4518   , piece_id=PFDXX6IYRUHREQNGMARYREHQ3T4IHFFR3ZCF4JQABLLZGICTQYRQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.756720481 (593h36m32.627803516s) [noise=true , address=145.79.227.22:4509   , piece_id=JPQ4KCU2PWNV3IKGAPCI5ZRX42W7JQEMYQJOC7ABJATOCX463CGA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.771933810 (593h36m32.643016855s) [noise=true , address=145.79.227.10:4401   , piece_id=7FJNWJIW72O3PGSBKNOHCKME4VP44M5V5D7LBFZ2UJJ7TAPQP2YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.774008941 (593h36m32.645091986s) [noise=true , address=145.79.227.29:4941   , piece_id=AP443WQDGBEJI6MXB7CYDDKUF5LZW3VKMF26NKK2SM23ESHWEBMQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.824307567 (593h36m32.695390602s) [noise=true , address=145.79.227.50:4373   , piece_id=NPXA4NQ2NSVQCX5BLTZNFUAGFTNZSSV35OVJHSHXYSUOBVVSEHUA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.915758335 (593h36m32.78684137s) [noise=true , address=77.90.31.15:4697     , piece_id=TU3RUHHCBOKBEZ6KZN6B2J5KJ5ZKAURKIYTRF4JR3YCW2ZTQC3YA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:32.930883967 (593h36m32.801967012s) [noise=true , address=77.90.31.21:4878     , piece_id=UXE3U4DZKVRCBX5XNF4VGZL62TKQBR3MVVS5276YASDUEMXBK7MQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.053179076 (593h36m32.924262112s) [noise=true , address=45.91.116.9:4264     , piece_id=CLWLL4BAONV3KRZMTTV33DKBN34BIGDQNDFSWOTBUEVC7Y6TEIZQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095502747 (593h36m32.966585782s) [noise=true , address=45.91.116.5:4299     , piece_id=BDO2IRABIKGLLFW3MLABHDGXENBHKLHASJGXTW5MOHBFUN7HWMSQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: <nil>
Dec 30 09:05:33.095525922 (593h36m32.966608957s) [seg_pos={0 0}, upload=520388]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:05:33.095818288 (593h36m32.966901323s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095826144 (593h36m32.966909179s) [noise=true , address=45.91.116.10:4083    , piece_id=RMNSLXMHFJOFMYBFV4T3D6D6TURQTZDVXBPHYQOZLAZT5HCAZKVA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.095853635 (593h36m32.96693668s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 30 09:05:33.095862242 (593h36m32.966945278s) [noise=true , address=45.91.116.8:4045     , piece_id=NEB6FKIX6SOMRBG2WJY6FBNSR3A7G6TKL2HGGAVROSTWIXHW6CYA, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096036795 (593h36m32.96711983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56196->45.91.116.13:4155: use of closed network connection
Dec 30 09:05:33.096043938 (593h36m32.967126983s) [noise=true , address=45.91.116.13:4155    , piece_id=M23T5XPGL6WV64UMW7OE6MCLQP35DWSM5B3EBL7YIOGXHWJVMDYQ, seg_pos={0 0}, upload=520388]: Long tail cancellation due to optimal threshold
Dec 30 09:05:33.096069176 (593h36m32.967152221s) [seg_pos={0 0}, upload=520388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:05:33.096170128 (593h36m32.967253173s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:05:33.096176110 (593h36m32.967259145s) [upload=520388]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:05:33.389556302 (593h36m33.260639347s) [upload=520388]: Done uploading segments...
Dec 30 09:05:33.389580217 (593h36m33.260663262s) [upload=520388]: Upload finished. err: <nil>
Duration: 30.003289362s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [...................X....X.....X.........X...X.X....X.X...X..]
duration: 
4.142391348s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:05:23 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.142423891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [...X...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...]
duration: 
6.006889783s
trace: 
https://jaeger.dev.storj.io/trace/747e3cb14f34091a
FAIL (2025-12-30 09:05:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA) context deadline exceeded
======LOGS======
ments...
Dec 30 09:02:31.667859845 (593h33m31.53894289s) [upload=520344]: Got next segment. Inline: false
Dec 30 09:02:31.667898849 (593h33m31.538981894s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:02:31.667906814 (593h33m31.538989849s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:02:31.911288441 (593h33m31.782371486s) [seg_pos={0 0}, upload=520344]: Begin upload segment...
Dec 30 09:02:31.911346743 (593h33m31.782429788s) [seg_pos={0 0}, upload=520344]: Done begin upload segment.
Dec 30 09:02:31.911383002 (593h33m31.782466047s) [noise=true , address=145.79.227.7:4293    , piece_id=RDEJMGXX72HIIU6JALZACV5KA4VZ3I4TAKAOU6XT7BSEEIRWZCUQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.911411536 (593h33m31.782494571s) [noise=true , address=145.79.227.29:4942   , piece_id=7AVNZKRYFLGL5V33ZLXKBVYWYQXQN5FEU5OOYNEER2AC3DZSEW7Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.911588753 (593h33m31.782671809s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.912264780 (593h33m31.783347825s) [noise=true , address=145.79.227.23:4546   , piece_id=FEPBCKM43NMMSCCAZWJE6IVJU4KWVLY2DULMBE2F7YEKTJLM7XZA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.912781033 (593h33m31.783864079s) [noise=true , address=145.79.227.18:4006   , piece_id=NPYRAQU7BPTAJPH2FBLACX5O4J36GJCETAH4Z6YU6WHRICOGRKVQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913222294 (593h33m31.784305329s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913229317 (593h33m31.784312352s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913694243 (593h33m31.784777278s) [noise=true , address=145.79.227.13:4803   , piece_id=VALIJXNJMELYU72H3L6WL2J3OBSFA2XYKQV7I6ALCVOAWPH3KIRA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914155191 (593h33m31.785238236s) [noise=true , address=77.90.31.11:4553     , piece_id=FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914429273 (593h33m31.785512318s) [noise=true , address=145.79.227.19:4079   , piece_id=63EP2O52E2RY4KJJRTA4LFIVFLD5QYGQKHAOKUJKHLMTYFYKRR3A, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914615036 (593h33m31.785698081s) [noise=true , address=77.90.31.14:4661     , piece_id=VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914938672 (593h33m31.786021707s) [noise=true , address=145.79.227.14:4837   , piece_id=44FB5TAUY67D4OSR7SK3RPS7NZGEVDXM44WHMYGTK3WMHLEVOOWA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.915074130 (593h33m31.786157175s) [noise=true , address=145.79.227.8:4331    , piece_id=UFY5OCZIYHFBC7RYMO22FNICZIFC7UFETQF45DULRCJF5KUYXSVA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.915549806 (593h33m31.786632841s) [noise=true , address=145.79.227.11:4730   , piece_id=KNNS7BDD4SCKWXY3KBY7DCGDMBWNFJBOBDN5ZSKXFYFZXB6SN24A, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.916085726 (593h33m31.787168781s) [noise=true , address=145.79.227.15:4873   , piece_id=T2XAEMNYTOLPCCHTGK3UXQY65TMSSMCM3NINPZGQENVZ4OPLQ4SA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.918195133 (593h33m31.789278168s) [noise=true , address=145.79.227.12:4765   , piece_id=3MN4RNU4D274ADTOCJJ4ITWCFWUBLEGMAD5Q7VMEJ5SDQIE6IKCQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:33.515920470 (593h33m33.387003515s) [upload=520344]: Next returned nil segment
Dec 30 09:02:33.515935207 (593h33m33.387018262s) [upload=520344]: Waiting for error group managing segments...
Dec 30 09:02:33.898381507 (593h33m33.769464552s) [noise=true , address=145.79.227.18:4006   , piece_id=NPYRAQU7BPTAJPH2FBLACX5O4J36GJCETAH4Z6YU6WHRICOGRKVQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.898450178 (593h33m33.769533223s) [noise=true , address=145.79.227.29:4942   , piece_id=7AVNZKRYFLGL5V33ZLXKBVYWYQXQN5FEU5OOYNEER2AC3DZSEW7Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.900956380 (593h33m33.772039425s) [noise=true , address=145.79.227.11:4730   , piece_id=KNNS7BDD4SCKWXY3KBY7DCGDMBWNFJBOBDN5ZSKXFYFZXB6SN24A, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.902207612 (593h33m33.773290657s) [noise=true , address=145.79.227.12:4765   , piece_id=3MN4RNU4D274ADTOCJJ4ITWCFWUBLEGMAD5Q7VMEJ5SDQIE6IKCQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.903765930 (593h33m33.774848965s) [noise=true , address=145.79.227.19:4079   , piece_id=63EP2O52E2RY4KJJRTA4LFIVFLD5QYGQKHAOKUJKHLMTYFYKRR3A, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.904393485 (593h33m33.77547652s) [noise=true , address=145.79.227.15:4873   , piece_id=T2XAEMNYTOLPCCHTGK3UXQY65TMSSMCM3NINPZGQENVZ4OPLQ4SA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.912669343 (593h33m33.783752378s) [noise=true , address=145.79.227.23:4546   , piece_id=FEPBCKM43NMMSCCAZWJE6IVJU4KWVLY2DULMBE2F7YEKTJLM7XZA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.927765218 (593h33m33.798848253s) [noise=true , address=145.79.227.7:4293    , piece_id=RDEJMGXX72HIIU6JALZACV5KA4VZ3I4TAKAOU6XT7BSEEIRWZCUQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.943681546 (593h33m33.814764591s) [noise=true , address=145.79.227.8:4331    , piece_id=UFY5OCZIYHFBC7RYMO22FNICZIFC7UFETQF45DULRCJF5KUYXSVA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.987038454 (593h33m33.858121499s) [noise=true , address=145.79.227.14:4837   , piece_id=44FB5TAUY67D4OSR7SK3RPS7NZGEVDXM44WHMYGTK3WMHLEVOOWA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.992443416 (593h33m33.863526451s) [noise=true , address=145.79.227.13:4803   , piece_id=VALIJXNJMELYU72H3L6WL2J3OBSFA2XYKQV7I6ALCVOAWPH3KIRA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.089020444 (593h33m33.960103489s) [noise=true , address=77.90.31.11:4553     , piece_id=FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.231364450 (593h33m34.102447495s) [noise=true , address=77.90.31.14:4661     , piece_id=VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.231391521 (593h33m34.102474566s) [seg_pos={0 0}, upload=520344]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:02:34.231603606 (593h33m34.102686651s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection
Dec 30 09:02:34.231680091 (593h33m34.102763136s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.231933874 (593h33m34.103016909s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection
Dec 30 09:02:34.231954163 (593h33m34.103037198s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.232042731 (593h33m34.103125766s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 30 09:02:34.232052450 (593h33m34.103135495s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.232073310 (593h33m34.103156355s) [seg_pos={0 0}, upload=520344]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:02:34.232157620 (593h33m34.103240655s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:02:34.232164383 (593h33m34.103247429s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:02:34.521600667 (593h33m34.392683712s) [upload=520344]: Done uploading segments...
Dec 30 09:02:34.521623249 (593h33m34.392706294s) [upload=520344]: Upload finished. err: <nil>
Duration: 6.006916484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...................X.....................X..................]
duration: 
38.150895223s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:04:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 38.150916612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [....................X....X.....X.........X...X.X....X.X...X.]
duration: 
3.882015703s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:04:40 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.882046271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.....................X....X.....X.........X...X.X....X.X...X]
duration: 
5.001216553s
trace: 
https://jaeger.dev.storj.io/trace/6c407bbbae47006e
FAIL (2025-12-30 09:04:22 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001239727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X...X...X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X..]
duration: 
7.384823745s
trace: 
https://jaeger.dev.storj.io/trace/747e3cb14f34091a
FAIL (2025-12-30 09:04:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA) context deadline exceeded
======LOGS======
ments...
Dec 30 09:02:31.667859845 (593h33m31.53894289s) [upload=520344]: Got next segment. Inline: false
Dec 30 09:02:31.667898849 (593h33m31.538981894s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 09:02:31.667906814 (593h33m31.538989849s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 09:02:31.911288441 (593h33m31.782371486s) [seg_pos={0 0}, upload=520344]: Begin upload segment...
Dec 30 09:02:31.911346743 (593h33m31.782429788s) [seg_pos={0 0}, upload=520344]: Done begin upload segment.
Dec 30 09:02:31.911383002 (593h33m31.782466047s) [noise=true , address=145.79.227.7:4293    , piece_id=RDEJMGXX72HIIU6JALZACV5KA4VZ3I4TAKAOU6XT7BSEEIRWZCUQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.911411536 (593h33m31.782494571s) [noise=true , address=145.79.227.29:4942   , piece_id=7AVNZKRYFLGL5V33ZLXKBVYWYQXQN5FEU5OOYNEER2AC3DZSEW7Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.911588753 (593h33m31.782671809s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.912264780 (593h33m31.783347825s) [noise=true , address=145.79.227.23:4546   , piece_id=FEPBCKM43NMMSCCAZWJE6IVJU4KWVLY2DULMBE2F7YEKTJLM7XZA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.912781033 (593h33m31.783864079s) [noise=true , address=145.79.227.18:4006   , piece_id=NPYRAQU7BPTAJPH2FBLACX5O4J36GJCETAH4Z6YU6WHRICOGRKVQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913222294 (593h33m31.784305329s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913229317 (593h33m31.784312352s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.913694243 (593h33m31.784777278s) [noise=true , address=145.79.227.13:4803   , piece_id=VALIJXNJMELYU72H3L6WL2J3OBSFA2XYKQV7I6ALCVOAWPH3KIRA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914155191 (593h33m31.785238236s) [noise=true , address=77.90.31.11:4553     , piece_id=FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914429273 (593h33m31.785512318s) [noise=true , address=145.79.227.19:4079   , piece_id=63EP2O52E2RY4KJJRTA4LFIVFLD5QYGQKHAOKUJKHLMTYFYKRR3A, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914615036 (593h33m31.785698081s) [noise=true , address=77.90.31.14:4661     , piece_id=VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.914938672 (593h33m31.786021707s) [noise=true , address=145.79.227.14:4837   , piece_id=44FB5TAUY67D4OSR7SK3RPS7NZGEVDXM44WHMYGTK3WMHLEVOOWA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.915074130 (593h33m31.786157175s) [noise=true , address=145.79.227.8:4331    , piece_id=UFY5OCZIYHFBC7RYMO22FNICZIFC7UFETQF45DULRCJF5KUYXSVA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.915549806 (593h33m31.786632841s) [noise=true , address=145.79.227.11:4730   , piece_id=KNNS7BDD4SCKWXY3KBY7DCGDMBWNFJBOBDN5ZSKXFYFZXB6SN24A, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.916085726 (593h33m31.787168781s) [noise=true , address=145.79.227.15:4873   , piece_id=T2XAEMNYTOLPCCHTGK3UXQY65TMSSMCM3NINPZGQENVZ4OPLQ4SA, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:31.918195133 (593h33m31.789278168s) [noise=true , address=145.79.227.12:4765   , piece_id=3MN4RNU4D274ADTOCJJ4ITWCFWUBLEGMAD5Q7VMEJ5SDQIE6IKCQ, seg_pos={0 0}, upload=520344]: Uploading piece...
Dec 30 09:02:33.515920470 (593h33m33.387003515s) [upload=520344]: Next returned nil segment
Dec 30 09:02:33.515935207 (593h33m33.387018262s) [upload=520344]: Waiting for error group managing segments...
Dec 30 09:02:33.898381507 (593h33m33.769464552s) [noise=true , address=145.79.227.18:4006   , piece_id=NPYRAQU7BPTAJPH2FBLACX5O4J36GJCETAH4Z6YU6WHRICOGRKVQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.898450178 (593h33m33.769533223s) [noise=true , address=145.79.227.29:4942   , piece_id=7AVNZKRYFLGL5V33ZLXKBVYWYQXQN5FEU5OOYNEER2AC3DZSEW7Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.900956380 (593h33m33.772039425s) [noise=true , address=145.79.227.11:4730   , piece_id=KNNS7BDD4SCKWXY3KBY7DCGDMBWNFJBOBDN5ZSKXFYFZXB6SN24A, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.902207612 (593h33m33.773290657s) [noise=true , address=145.79.227.12:4765   , piece_id=3MN4RNU4D274ADTOCJJ4ITWCFWUBLEGMAD5Q7VMEJ5SDQIE6IKCQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.903765930 (593h33m33.774848965s) [noise=true , address=145.79.227.19:4079   , piece_id=63EP2O52E2RY4KJJRTA4LFIVFLD5QYGQKHAOKUJKHLMTYFYKRR3A, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.904393485 (593h33m33.77547652s) [noise=true , address=145.79.227.15:4873   , piece_id=T2XAEMNYTOLPCCHTGK3UXQY65TMSSMCM3NINPZGQENVZ4OPLQ4SA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.912669343 (593h33m33.783752378s) [noise=true , address=145.79.227.23:4546   , piece_id=FEPBCKM43NMMSCCAZWJE6IVJU4KWVLY2DULMBE2F7YEKTJLM7XZA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.927765218 (593h33m33.798848253s) [noise=true , address=145.79.227.7:4293    , piece_id=RDEJMGXX72HIIU6JALZACV5KA4VZ3I4TAKAOU6XT7BSEEIRWZCUQ, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.943681546 (593h33m33.814764591s) [noise=true , address=145.79.227.8:4331    , piece_id=UFY5OCZIYHFBC7RYMO22FNICZIFC7UFETQF45DULRCJF5KUYXSVA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.987038454 (593h33m33.858121499s) [noise=true , address=145.79.227.14:4837   , piece_id=44FB5TAUY67D4OSR7SK3RPS7NZGEVDXM44WHMYGTK3WMHLEVOOWA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:33.992443416 (593h33m33.863526451s) [noise=true , address=145.79.227.13:4803   , piece_id=VALIJXNJMELYU72H3L6WL2J3OBSFA2XYKQV7I6ALCVOAWPH3KIRA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.089020444 (593h33m33.960103489s) [noise=true , address=77.90.31.11:4553     , piece_id=FOZBBMKPVWOZA7QWY6APEQAN3BJ64K7B5KMUGFMN7LDBY6TCKV4Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.231364450 (593h33m34.102447495s) [noise=true , address=77.90.31.14:4661     , piece_id=VJRWZJ42DZNSESGOYHA6VTB3NOOAM7JSAHX6RBFOYADQKUDSXBEA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: <nil>
Dec 30 09:02:34.231391521 (593h33m34.102474566s) [seg_pos={0 0}, upload=520344]: Segment reached optimal threshold of 13 pieces.
Dec 30 09:02:34.231603606 (593h33m34.102686651s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37576->77.90.31.15:4697: use of closed network connection
Dec 30 09:02:34.231680091 (593h33m34.102763136s) [noise=true , address=77.90.31.15:4697     , piece_id=BZNND4D5IETZAITDXZDVLYOBMAJINETVYNFLASNZPB6YMTZWN7RA, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.231933874 (593h33m34.103016909s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33044->45.91.116.6:4975: use of closed network connection
Dec 30 09:02:34.231954163 (593h33m34.103037198s) [noise=true , address=45.91.116.6:4975     , piece_id=EEJLAEH6DPGCB5NTXDU63PRHELVGHUYP7O6GFMTBE7IV2MT3TMXA, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.232042731 (593h33m34.103125766s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 30 09:02:34.232052450 (593h33m34.103135495s) [noise=true , address=45.91.116.14:4192    , piece_id=OMSJC4NYQOOV2VJZY65JUQAQ7HZANL5SEBMUT3O66OXQJJFJEG4Q, seg_pos={0 0}, upload=520344]: Long tail cancellation due to optimal threshold
Dec 30 09:02:34.232073310 (593h33m34.103156355s) [seg_pos={0 0}, upload=520344]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 09:02:34.232157620 (593h33m34.103240655s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 09:02:34.232164383 (593h33m34.103247429s) [upload=520344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 09:02:34.521600667 (593h33m34.392683712s) [upload=520344]: Done uploading segments...
Dec 30 09:02:34.521623249 (593h33m34.392706294s) [upload=520344]: Upload finished. err: <nil>
Duration: 7.384849603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [....................X.....................X.................]
duration: 
39.334532912s
trace: 
https://jaeger.dev.storj.io/trace/e14e3f77cfefb30
FAIL (2025-12-30 09:04:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece...
Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment
Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments...
Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964     , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956     , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987   , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955   , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683     , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594     , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713     , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055   , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186    , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902     , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202   , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765   , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934   , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil>
Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection
Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707    , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF
Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496   , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection
Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170   , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled
Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881   , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold
Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments...
Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil>
Duration: 39.334558301s core uplink us-select-1 us1-placement-20