Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 336800 reports (Page 129 of 6736)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [..XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...X.X..]
duration: 
8.775895644s
trace: 
https://jaeger.dev.storj.io/trace/197be3340a354980
FAIL (2025-12-27 17:58:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 17:57:13.184599342 (530h28m13.055682377s) [upload=465232]: Starting upload
Dec 27 17:57:13.184658033 (530h28m13.055741068s) [upload=465232]: Done starting upload
Dec 27 17:57:13.184817246 (530h28m13.055900281s) [upload=465232]: Uploading segments...
Dec 27 17:57:13.184942936 (530h28m13.056025981s) [upload=465232]: Got next segment. Inline: false
Dec 27 17:57:13.184989615 (530h28m13.05607265s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:57:13.185007007 (530h28m13.056090052s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:57:13.427654601 (530h28m13.298737646s) [seg_pos={0 0}, upload=465232]: Begin upload segment...
Dec 27 17:57:13.427711458 (530h28m13.298794493s) [seg_pos={0 0}, upload=465232]: Done begin upload segment.
Dec 27 17:57:13.427762206 (530h28m13.298845241s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430025846 (530h28m13.301108881s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430404236 (530h28m13.301487272s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430775874 (530h28m13.30185891s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431044876 (530h28m13.302127911s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431310402 (530h28m13.302393437s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431581779 (530h28m13.302664824s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431895326 (530h28m13.302978361s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432205747 (530h28m13.303288782s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432541516 (530h28m13.303624551s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432850905 (530h28m13.30393394s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433270173 (530h28m13.304353208s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433732093 (530h28m13.304815138s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434215313 (530h28m13.305298348s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434663537 (530h28m13.305746582s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.435105348 (530h28m13.306188383s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:15.137992939 (530h28m15.009075974s) [upload=465232]: Next returned nil segment
Dec 27 17:57:15.138003629 (530h28m15.009086674s) [upload=465232]: Waiting for error group managing segments...
Dec 27 17:57:15.419760623 (530h28m15.290843658s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433368314 (530h28m15.304451349s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433550361 (530h28m15.304633406s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.442895756 (530h28m15.313978791s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.444442040 (530h28m15.315525085s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.448760884 (530h28m15.319843929s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.467441845 (530h28m15.33852488s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.695069488 (530h28m15.566152523s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.734043161 (530h28m15.605126196s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.757150511 (530h28m15.628233556s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.799710672 (530h28m15.670793718s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.859866690 (530h28m15.730949735s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415474002 (530h28m16.286557037s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415493068 (530h28m16.286576103s) [seg_pos={0 0}, upload=465232]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:57:16.415709219 (530h28m16.286792264s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415716975 (530h28m16.28680002s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415722776 (530h28m16.286805821s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415730580 (530h28m16.286813616s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415878281 (530h28m16.286961317s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415885725 (530h28m16.28696876s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415905123 (530h28m16.286988168s) [seg_pos={0 0}, upload=465232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:57:16.415973994 (530h28m16.287057039s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:57:16.415979875 (530h28m16.28706291s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:57:16.701627738 (530h28m16.572710783s) [upload=465232]: Done uploading segments...
Dec 27 17:57:16.701652996 (530h28m16.572736032s) [upload=465232]: Upload finished. err: <nil>
Duration: 8.775925761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...X.X.]
duration: 
16.023310275s
trace: 
https://jaeger.dev.storj.io/trace/197be3340a354980
FAIL (2025-12-27 17:57:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 17:57:13.184599342 (530h28m13.055682377s) [upload=465232]: Starting upload
Dec 27 17:57:13.184658033 (530h28m13.055741068s) [upload=465232]: Done starting upload
Dec 27 17:57:13.184817246 (530h28m13.055900281s) [upload=465232]: Uploading segments...
Dec 27 17:57:13.184942936 (530h28m13.056025981s) [upload=465232]: Got next segment. Inline: false
Dec 27 17:57:13.184989615 (530h28m13.05607265s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:57:13.185007007 (530h28m13.056090052s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:57:13.427654601 (530h28m13.298737646s) [seg_pos={0 0}, upload=465232]: Begin upload segment...
Dec 27 17:57:13.427711458 (530h28m13.298794493s) [seg_pos={0 0}, upload=465232]: Done begin upload segment.
Dec 27 17:57:13.427762206 (530h28m13.298845241s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430025846 (530h28m13.301108881s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430404236 (530h28m13.301487272s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430775874 (530h28m13.30185891s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431044876 (530h28m13.302127911s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431310402 (530h28m13.302393437s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431581779 (530h28m13.302664824s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431895326 (530h28m13.302978361s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432205747 (530h28m13.303288782s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432541516 (530h28m13.303624551s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432850905 (530h28m13.30393394s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433270173 (530h28m13.304353208s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433732093 (530h28m13.304815138s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434215313 (530h28m13.305298348s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434663537 (530h28m13.305746582s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.435105348 (530h28m13.306188383s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:15.137992939 (530h28m15.009075974s) [upload=465232]: Next returned nil segment
Dec 27 17:57:15.138003629 (530h28m15.009086674s) [upload=465232]: Waiting for error group managing segments...
Dec 27 17:57:15.419760623 (530h28m15.290843658s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433368314 (530h28m15.304451349s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433550361 (530h28m15.304633406s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.442895756 (530h28m15.313978791s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.444442040 (530h28m15.315525085s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.448760884 (530h28m15.319843929s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.467441845 (530h28m15.33852488s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.695069488 (530h28m15.566152523s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.734043161 (530h28m15.605126196s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.757150511 (530h28m15.628233556s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.799710672 (530h28m15.670793718s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.859866690 (530h28m15.730949735s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415474002 (530h28m16.286557037s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415493068 (530h28m16.286576103s) [seg_pos={0 0}, upload=465232]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:57:16.415709219 (530h28m16.286792264s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415716975 (530h28m16.28680002s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415722776 (530h28m16.286805821s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415730580 (530h28m16.286813616s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415878281 (530h28m16.286961317s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415885725 (530h28m16.28696876s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415905123 (530h28m16.286988168s) [seg_pos={0 0}, upload=465232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:57:16.415973994 (530h28m16.287057039s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:57:16.415979875 (530h28m16.28706291s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:57:16.701627738 (530h28m16.572710783s) [upload=465232]: Done uploading segments...
Dec 27 17:57:16.701652996 (530h28m16.572736032s) [upload=465232]: Upload finished. err: <nil>
Duration: 16.023329222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...X.X]
duration: 
30.002736236s
trace: 
https://jaeger.dev.storj.io/trace/197be3340a354980
FAIL (2025-12-27 17:57:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 17:57:13.184599342 (530h28m13.055682377s) [upload=465232]: Starting upload
Dec 27 17:57:13.184658033 (530h28m13.055741068s) [upload=465232]: Done starting upload
Dec 27 17:57:13.184817246 (530h28m13.055900281s) [upload=465232]: Uploading segments...
Dec 27 17:57:13.184942936 (530h28m13.056025981s) [upload=465232]: Got next segment. Inline: false
Dec 27 17:57:13.184989615 (530h28m13.05607265s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:57:13.185007007 (530h28m13.056090052s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:57:13.427654601 (530h28m13.298737646s) [seg_pos={0 0}, upload=465232]: Begin upload segment...
Dec 27 17:57:13.427711458 (530h28m13.298794493s) [seg_pos={0 0}, upload=465232]: Done begin upload segment.
Dec 27 17:57:13.427762206 (530h28m13.298845241s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430025846 (530h28m13.301108881s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430404236 (530h28m13.301487272s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.430775874 (530h28m13.30185891s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431044876 (530h28m13.302127911s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431310402 (530h28m13.302393437s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431581779 (530h28m13.302664824s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.431895326 (530h28m13.302978361s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432205747 (530h28m13.303288782s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432541516 (530h28m13.303624551s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.432850905 (530h28m13.30393394s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433270173 (530h28m13.304353208s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.433732093 (530h28m13.304815138s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434215313 (530h28m13.305298348s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.434663537 (530h28m13.305746582s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:13.435105348 (530h28m13.306188383s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Uploading piece...
Dec 27 17:57:15.137992939 (530h28m15.009075974s) [upload=465232]: Next returned nil segment
Dec 27 17:57:15.138003629 (530h28m15.009086674s) [upload=465232]: Waiting for error group managing segments...
Dec 27 17:57:15.419760623 (530h28m15.290843658s) [noise=true , address=145.79.227.15:4874   , piece_id=TP36XWITTYGCO4ZT2E7FV66P25B6N35W3QLO5YCA5P4LOJMWMS6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433368314 (530h28m15.304451349s) [noise=true , address=145.79.227.28:4727   , piece_id=CZRKYGGRTSC6A2CMDK2CPSBYLDXCXPF3FFYGWABU4CGXLQDFZYXA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.433550361 (530h28m15.304633406s) [noise=true , address=145.79.227.26:4655   , piece_id=S3M7IPFHMLZTWHV6A5PPCDHHYJXDPQ57HGGD3OZ74I3KAGTKOPXQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.442895756 (530h28m15.313978791s) [noise=true , address=145.79.227.32:4836   , piece_id=BDNKABOSI43OFI44MQAC4LL7LYYSNYD4KAOGP3ZJPZNVJW4W7NDQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.444442040 (530h28m15.315525085s) [noise=true , address=145.79.227.52:4446   , piece_id=MIBMOCCEFUXL7BAYXHMTEBD4WZKVEHJGKX6RNDNHKVYTCGDBZGDA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.448760884 (530h28m15.319843929s) [noise=true , address=145.79.227.47:4267   , piece_id=DAHPGD4K2QXLRHCRMVI4ZT6KA4D25MSI7KM5ZUMUG7HDTFDPEV3Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.467441845 (530h28m15.33852488s) [noise=true , address=145.79.227.45:4230   , piece_id=SBYG7UOYQC5ATFOGVFTCWN3QXIES3Y2W5UN5CS6TKJKIJQX5QMRQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.695069488 (530h28m15.566152523s) [noise=true , address=45.91.116.9:4261     , piece_id=Z66H4CB5J74RRV37VZDSUASVH4TITV776AC526ATJFPT3C3XPWCQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.734043161 (530h28m15.605126196s) [noise=true , address=45.91.116.5:4297     , piece_id=5NX5ELATDE33CVKOFUBLCQQEEGNF7U2PJHWHERLXSF5FFMSLDSFA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.757150511 (530h28m15.628233556s) [noise=true , address=45.91.116.11:4118    , piece_id=QCDKVY72MK3VDCYY27M2OXJJN2BBQAJP6KAIK444YOFJ6T67NBTQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.799710672 (530h28m15.670793718s) [noise=true , address=45.91.116.8:4047     , piece_id=Z5JYHOEH554V4LOJNVNODI5OS7PUKJNPUBJNO4ECLIBJODOUFZSQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:15.859866690 (530h28m15.730949735s) [noise=true , address=45.91.116.10:4081    , piece_id=MJIKB5VLUYOLYMZEIUX3LCLCSX2EASAF6OHO37T2EMBAXB23GU6A, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415474002 (530h28m16.286557037s) [noise=true , address=77.90.31.17:4773     , piece_id=DUADYFVDLTY4ZFTMHOGHL4U5T3PW4OFJQFWRPPJY67CADXDP6SNA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: <nil>
Dec 27 17:57:16.415493068 (530h28m16.286576103s) [seg_pos={0 0}, upload=465232]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:57:16.415709219 (530h28m16.286792264s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415716975 (530h28m16.28680002s) [noise=true , address=45.91.116.12:4225    , piece_id=7GVLAWHYR3C5T2QYCLDIYCW74KEJKGVAEXIA4EBII3V2WABTRAIA, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415722776 (530h28m16.286805821s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415730580 (530h28m16.286813616s) [noise=true , address=45.91.116.13:4155    , piece_id=SIMQ726ATU4JSI6CVJRAJMFGIHFTQHT4UEVGWKLMYNDG7U5PPW4Q, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415878281 (530h28m16.286961317s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 27 17:57:16.415885725 (530h28m16.28696876s) [noise=true , address=77.90.31.12:4589     , piece_id=RU7POIVZYSDRGVEE3XHKT5WESFI6GK4NAFKMF6LAYIC2SEX35WEQ, seg_pos={0 0}, upload=465232]: Long tail cancellation due to optimal threshold
Dec 27 17:57:16.415905123 (530h28m16.286988168s) [seg_pos={0 0}, upload=465232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:57:16.415973994 (530h28m16.287057039s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:57:16.415979875 (530h28m16.28706291s) [upload=465232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:57:16.701627738 (530h28m16.572710783s) [upload=465232]: Done uploading segments...
Dec 27 17:57:16.701652996 (530h28m16.572736032s) [upload=465232]: Upload finished. err: <nil>
Duration: 30.002773156s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................................X..................]
duration: 
3.499436728s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:57:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.499465674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................................X.................]
duration: 
3.485252531s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:57:16 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.485281285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................................X................]
duration: 
3.377350646s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:56:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.377382587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [.X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...X.]
duration: 
8.105932298s
trace: 
https://jaeger.dev.storj.io/trace/fac0a6bfd105423
FAIL (2025-12-27 17:56:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA) context deadline exceeded
======LOGS======
Dec 27 17:55:51.498869122 (530h26m51.369952167s) [upload=465214]: Starting upload
Dec 27 17:55:51.498945999 (530h26m51.370029034s) [upload=465214]: Done starting upload
Dec 27 17:55:51.499006404 (530h26m51.370089449s) [upload=465214]: Uploading segments...
Dec 27 17:55:51.499242734 (530h26m51.370325769s) [upload=465214]: Got next segment. Inline: false
Dec 27 17:55:51.499286337 (530h26m51.370369372s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:55:51.499292128 (530h26m51.370375173s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:55:51.739591398 (530h26m51.610674433s) [seg_pos={0 0}, upload=465214]: Begin upload segment...
Dec 27 17:55:51.739641103 (530h26m51.610724138s) [seg_pos={0 0}, upload=465214]: Done begin upload segment.
Dec 27 17:55:51.739667834 (530h26m51.610750869s) [noise=true , address=45.91.116.5:4299     , piece_id=3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.740937451 (530h26m51.612020486s) [noise=true , address=145.79.227.10:4402   , piece_id=VV4RECQA74QRVQWRXTNURCUQOIKSCCVNXKUEEUDFTDJPRRCF22PQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.741283090 (530h26m51.612366145s) [noise=true , address=145.79.227.12:4765   , piece_id=CT2PNNZ27XNS6BJVMQQVHJTPE2POFWU4XUF2NPMNF77CQTACQPIQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.741923028 (530h26m51.613006063s) [noise=true , address=145.79.227.8:4329    , piece_id=VRGTBSGUJRTD4XB3O2PJA4UQNDANO4AOTZUPIEVGVYSHNBJBPZ4Q, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.742380299 (530h26m51.613463344s) [noise=true , address=45.91.116.8:4045     , piece_id=IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.742710047 (530h26m51.613793092s) [noise=true , address=145.79.227.42:4123   , piece_id=E5PQJG7JTERWTBJ5QYD4L67DLLJNIMBFLQP37VMH2YEBQWRH5AVA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743130898 (530h26m51.614213943s) [noise=true , address=145.79.227.2:4114    , piece_id=ENC7LR5MG2AR6XFJWIWYTGYJKXQNIFDDBO3M5OK5BJN5CNFH6LCA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743469563 (530h26m51.614552608s) [noise=true , address=145.79.227.39:4014   , piece_id=QSXGIHN6UKCVMAKJOAEMGSMQCMTXATMQBTI5E67PFOZJ53W6LF6Q, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743739297 (530h26m51.614822332s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743880145 (530h26m51.61496319s) [noise=true , address=145.79.227.9:4366    , piece_id=PAYZD4C56YEDRGSRVMPSJPCU54ABQITU64PYZPF5NVKDFCRMHWYQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.744215383 (530h26m51.615298428s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.744848639 (530h26m51.615931684s) [noise=true , address=145.79.227.48:4303   , piece_id=3RBDDVYA3RDRARFRICGJO347GNZ2DVI7ZM37AXCDP6Z7O3AARMPA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745030886 (530h26m51.616113932s) [noise=true , address=145.79.227.50:4374   , piece_id=TYSCJDUFHRXM36FJHFQWPCEWCMKOTC22UWJG7GXJSGR7VLCUQYBA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745522683 (530h26m51.616605718s) [noise=true , address=145.79.227.14:4839   , piece_id=OWWJ6NNG2FWFP32UHJ6CR3PXQ3BH2TPOWQHC4FZVKLJ5TT3GDQTA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745979192 (530h26m51.617062227s) [noise=true , address=145.79.227.36:4517   , piece_id=P3FCUXOXHRTFT5QIUW3ZMWUXSRDIOEOF7APKXKUVGU2RQEVEPGLQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.746392399 (530h26m51.617475444s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:53.456154731 (530h26m53.327237776s) [upload=465214]: Next returned nil segment
Dec 27 17:55:53.456174307 (530h26m53.327257342s) [upload=465214]: Waiting for error group managing segments...
Dec 27 17:55:53.740473031 (530h26m53.611556076s) [noise=true , address=145.79.227.9:4366    , piece_id=PAYZD4C56YEDRGSRVMPSJPCU54ABQITU64PYZPF5NVKDFCRMHWYQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.746110837 (530h26m53.617193872s) [noise=true , address=145.79.227.10:4402   , piece_id=VV4RECQA74QRVQWRXTNURCUQOIKSCCVNXKUEEUDFTDJPRRCF22PQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.752545670 (530h26m53.623628715s) [noise=true , address=145.79.227.42:4123   , piece_id=E5PQJG7JTERWTBJ5QYD4L67DLLJNIMBFLQP37VMH2YEBQWRH5AVA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.759396936 (530h26m53.630479981s) [noise=true , address=145.79.227.2:4114    , piece_id=ENC7LR5MG2AR6XFJWIWYTGYJKXQNIFDDBO3M5OK5BJN5CNFH6LCA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.760073744 (530h26m53.631156779s) [noise=true , address=145.79.227.8:4329    , piece_id=VRGTBSGUJRTD4XB3O2PJA4UQNDANO4AOTZUPIEVGVYSHNBJBPZ4Q, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.763309756 (530h26m53.634392801s) [noise=true , address=145.79.227.12:4765   , piece_id=CT2PNNZ27XNS6BJVMQQVHJTPE2POFWU4XUF2NPMNF77CQTACQPIQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.763965245 (530h26m53.63504828s) [noise=true , address=145.79.227.39:4014   , piece_id=QSXGIHN6UKCVMAKJOAEMGSMQCMTXATMQBTI5E67PFOZJ53W6LF6Q, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.767492210 (530h26m53.638575255s) [noise=true , address=145.79.227.48:4303   , piece_id=3RBDDVYA3RDRARFRICGJO347GNZ2DVI7ZM37AXCDP6Z7O3AARMPA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.768133983 (530h26m53.639217028s) [noise=true , address=145.79.227.36:4517   , piece_id=P3FCUXOXHRTFT5QIUW3ZMWUXSRDIOEOF7APKXKUVGU2RQEVEPGLQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.770045051 (530h26m53.641128096s) [noise=true , address=145.79.227.50:4374   , piece_id=TYSCJDUFHRXM36FJHFQWPCEWCMKOTC22UWJG7GXJSGR7VLCUQYBA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.890939645 (530h26m53.76202269s) [noise=true , address=145.79.227.14:4839   , piece_id=OWWJ6NNG2FWFP32UHJ6CR3PXQ3BH2TPOWQHC4FZVKLJ5TT3GDQTA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.978380618 (530h26m53.849463663s) [noise=true , address=45.91.116.5:4299     , piece_id=3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:54.025835260 (530h26m53.896918295s) [noise=true , address=45.91.116.8:4045     , piece_id=IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:54.025861640 (530h26m53.896944675s) [seg_pos={0 0}, upload=465214]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:55:54.026106146 (530h26m53.897189181s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026116847 (530h26m53.897199882s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026129270 (530h26m53.897212315s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026137796 (530h26m53.897220841s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026356232 (530h26m53.897439277s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026365300 (530h26m53.897448355s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026383935 (530h26m53.89746697s) [seg_pos={0 0}, upload=465214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:55:54.026548308 (530h26m53.897631343s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:55:54.026553698 (530h26m53.897636733s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:55:54.311430141 (530h26m54.182513186s) [upload=465214]: Done uploading segments...
Dec 27 17:55:54.311450410 (530h26m54.182533455s) [upload=465214]: Upload finished. err: <nil>
Duration: 8.105965572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [..X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...X]
duration: 
30.003815761s
trace: 
https://jaeger.dev.storj.io/trace/fac0a6bfd105423
FAIL (2025-12-27 17:56:20 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA) context deadline exceeded
======LOGS======
Dec 27 17:55:51.498869122 (530h26m51.369952167s) [upload=465214]: Starting upload
Dec 27 17:55:51.498945999 (530h26m51.370029034s) [upload=465214]: Done starting upload
Dec 27 17:55:51.499006404 (530h26m51.370089449s) [upload=465214]: Uploading segments...
Dec 27 17:55:51.499242734 (530h26m51.370325769s) [upload=465214]: Got next segment. Inline: false
Dec 27 17:55:51.499286337 (530h26m51.370369372s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:55:51.499292128 (530h26m51.370375173s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:55:51.739591398 (530h26m51.610674433s) [seg_pos={0 0}, upload=465214]: Begin upload segment...
Dec 27 17:55:51.739641103 (530h26m51.610724138s) [seg_pos={0 0}, upload=465214]: Done begin upload segment.
Dec 27 17:55:51.739667834 (530h26m51.610750869s) [noise=true , address=45.91.116.5:4299     , piece_id=3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.740937451 (530h26m51.612020486s) [noise=true , address=145.79.227.10:4402   , piece_id=VV4RECQA74QRVQWRXTNURCUQOIKSCCVNXKUEEUDFTDJPRRCF22PQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.741283090 (530h26m51.612366145s) [noise=true , address=145.79.227.12:4765   , piece_id=CT2PNNZ27XNS6BJVMQQVHJTPE2POFWU4XUF2NPMNF77CQTACQPIQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.741923028 (530h26m51.613006063s) [noise=true , address=145.79.227.8:4329    , piece_id=VRGTBSGUJRTD4XB3O2PJA4UQNDANO4AOTZUPIEVGVYSHNBJBPZ4Q, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.742380299 (530h26m51.613463344s) [noise=true , address=45.91.116.8:4045     , piece_id=IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.742710047 (530h26m51.613793092s) [noise=true , address=145.79.227.42:4123   , piece_id=E5PQJG7JTERWTBJ5QYD4L67DLLJNIMBFLQP37VMH2YEBQWRH5AVA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743130898 (530h26m51.614213943s) [noise=true , address=145.79.227.2:4114    , piece_id=ENC7LR5MG2AR6XFJWIWYTGYJKXQNIFDDBO3M5OK5BJN5CNFH6LCA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743469563 (530h26m51.614552608s) [noise=true , address=145.79.227.39:4014   , piece_id=QSXGIHN6UKCVMAKJOAEMGSMQCMTXATMQBTI5E67PFOZJ53W6LF6Q, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743739297 (530h26m51.614822332s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.743880145 (530h26m51.61496319s) [noise=true , address=145.79.227.9:4366    , piece_id=PAYZD4C56YEDRGSRVMPSJPCU54ABQITU64PYZPF5NVKDFCRMHWYQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.744215383 (530h26m51.615298428s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.744848639 (530h26m51.615931684s) [noise=true , address=145.79.227.48:4303   , piece_id=3RBDDVYA3RDRARFRICGJO347GNZ2DVI7ZM37AXCDP6Z7O3AARMPA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745030886 (530h26m51.616113932s) [noise=true , address=145.79.227.50:4374   , piece_id=TYSCJDUFHRXM36FJHFQWPCEWCMKOTC22UWJG7GXJSGR7VLCUQYBA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745522683 (530h26m51.616605718s) [noise=true , address=145.79.227.14:4839   , piece_id=OWWJ6NNG2FWFP32UHJ6CR3PXQ3BH2TPOWQHC4FZVKLJ5TT3GDQTA, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.745979192 (530h26m51.617062227s) [noise=true , address=145.79.227.36:4517   , piece_id=P3FCUXOXHRTFT5QIUW3ZMWUXSRDIOEOF7APKXKUVGU2RQEVEPGLQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:51.746392399 (530h26m51.617475444s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Uploading piece...
Dec 27 17:55:53.456154731 (530h26m53.327237776s) [upload=465214]: Next returned nil segment
Dec 27 17:55:53.456174307 (530h26m53.327257342s) [upload=465214]: Waiting for error group managing segments...
Dec 27 17:55:53.740473031 (530h26m53.611556076s) [noise=true , address=145.79.227.9:4366    , piece_id=PAYZD4C56YEDRGSRVMPSJPCU54ABQITU64PYZPF5NVKDFCRMHWYQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.746110837 (530h26m53.617193872s) [noise=true , address=145.79.227.10:4402   , piece_id=VV4RECQA74QRVQWRXTNURCUQOIKSCCVNXKUEEUDFTDJPRRCF22PQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.752545670 (530h26m53.623628715s) [noise=true , address=145.79.227.42:4123   , piece_id=E5PQJG7JTERWTBJ5QYD4L67DLLJNIMBFLQP37VMH2YEBQWRH5AVA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.759396936 (530h26m53.630479981s) [noise=true , address=145.79.227.2:4114    , piece_id=ENC7LR5MG2AR6XFJWIWYTGYJKXQNIFDDBO3M5OK5BJN5CNFH6LCA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.760073744 (530h26m53.631156779s) [noise=true , address=145.79.227.8:4329    , piece_id=VRGTBSGUJRTD4XB3O2PJA4UQNDANO4AOTZUPIEVGVYSHNBJBPZ4Q, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.763309756 (530h26m53.634392801s) [noise=true , address=145.79.227.12:4765   , piece_id=CT2PNNZ27XNS6BJVMQQVHJTPE2POFWU4XUF2NPMNF77CQTACQPIQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.763965245 (530h26m53.63504828s) [noise=true , address=145.79.227.39:4014   , piece_id=QSXGIHN6UKCVMAKJOAEMGSMQCMTXATMQBTI5E67PFOZJ53W6LF6Q, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.767492210 (530h26m53.638575255s) [noise=true , address=145.79.227.48:4303   , piece_id=3RBDDVYA3RDRARFRICGJO347GNZ2DVI7ZM37AXCDP6Z7O3AARMPA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.768133983 (530h26m53.639217028s) [noise=true , address=145.79.227.36:4517   , piece_id=P3FCUXOXHRTFT5QIUW3ZMWUXSRDIOEOF7APKXKUVGU2RQEVEPGLQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.770045051 (530h26m53.641128096s) [noise=true , address=145.79.227.50:4374   , piece_id=TYSCJDUFHRXM36FJHFQWPCEWCMKOTC22UWJG7GXJSGR7VLCUQYBA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.890939645 (530h26m53.76202269s) [noise=true , address=145.79.227.14:4839   , piece_id=OWWJ6NNG2FWFP32UHJ6CR3PXQ3BH2TPOWQHC4FZVKLJ5TT3GDQTA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:53.978380618 (530h26m53.849463663s) [noise=true , address=45.91.116.5:4299     , piece_id=3EGDMFY6NL56RBCUTCB4OPV4SNUVANBAR73F2OGUQOLZBKYPMFSA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:54.025835260 (530h26m53.896918295s) [noise=true , address=45.91.116.8:4045     , piece_id=IVU22QEVXCXRWMZQ77T2NBKCRWSJQ5GNVEMNMTTYEUGDCZ5S5AVQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: <nil>
Dec 27 17:55:54.025861640 (530h26m53.896944675s) [seg_pos={0 0}, upload=465214]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:55:54.026106146 (530h26m53.897189181s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026116847 (530h26m53.897199882s) [noise=true , address=45.91.116.10:4081    , piece_id=COQSKNAMVPBK23HXCL42NGF7T5MXCVDOU4LZAXBQMSWWB6RE5RNQ, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026129270 (530h26m53.897212315s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026137796 (530h26m53.897220841s) [noise=true , address=45.91.116.7:4010     , piece_id=R6ESZBRJEHHIMHTR36ODIXW5RM6ON46TBSNXQYRWRJ7SBOJ6HFTQ, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026356232 (530h26m53.897439277s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 17:55:54.026365300 (530h26m53.897448355s) [noise=true , address=145.79.227.40:4049   , piece_id=74YFJ2HUVAJMBE4FHF4237NG5M2A7UNJ2EUOGQKEQD74LGP46JUA, seg_pos={0 0}, upload=465214]: Long tail cancellation due to optimal threshold
Dec 27 17:55:54.026383935 (530h26m53.89746697s) [seg_pos={0 0}, upload=465214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:55:54.026548308 (530h26m53.897631343s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:55:54.026553698 (530h26m53.897636733s) [upload=465214]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:55:54.311430141 (530h26m54.182513186s) [upload=465214]: Done uploading segments...
Dec 27 17:55:54.311450410 (530h26m54.182533455s) [upload=465214]: Upload finished. err: <nil>
Duration: 30.003840088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............................................X...............]
duration: 
3.514679265s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:55:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.514710594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X...]
duration: 
11.172395921s
trace: 
https://jaeger.dev.storj.io/trace/7f227738e815068d
FAIL (2025-12-27 17:55:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
loading segments...
Dec 27 17:52:17.271669397 (530h23m17.142752442s) [upload=465160]: Got next segment. Inline: false
Dec 27 17:52:17.271700696 (530h23m17.142783741s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:52:17.271705495 (530h23m17.14278853s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:52:17.516501970 (530h23m17.387585005s) [seg_pos={0 0}, upload=465160]: Begin upload segment...
Dec 27 17:52:17.516548298 (530h23m17.387631333s) [seg_pos={0 0}, upload=465160]: Done begin upload segment.
Dec 27 17:52:17.516573536 (530h23m17.387656571s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.516647507 (530h23m17.387730552s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517273529 (530h23m17.388356564s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517671336 (530h23m17.388754381s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518066639 (530h23m17.389149684s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518520123 (530h23m17.389603158s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518955682 (530h23m17.390038717s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520080474 (530h23m17.391163509s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520388761 (530h23m17.391471796s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521286019 (530h23m17.392369054s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521940235 (530h23m17.39302327s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.522506884 (530h23m17.393589919s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.523750652 (530h23m17.394833687s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.524965336 (530h23m17.396048371s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.526192752 (530h23m17.397275798s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.528015083 (530h23m17.399098118s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:19.236153162 (530h23m19.107236197s) [upload=465160]: Next returned nil segment
Dec 27 17:52:19.236174633 (530h23m19.107257668s) [upload=465160]: Waiting for error group managing segments...
Dec 27 17:52:19.531582360 (530h23m19.402665405s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.531965430 (530h23m19.403048465s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.536525944 (530h23m19.407608989s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540017642 (530h23m19.411100687s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540394209 (530h23m19.411477244s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540740749 (530h23m19.411823784s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556124372 (530h23m19.427207407s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556473507 (530h23m19.427556552s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.559846770 (530h23m19.430929816s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.775170735 (530h23m19.64625378s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.793054708 (530h23m19.664137753s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.831761974 (530h23m19.702845009s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880923065 (530h23m19.7520061s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880944676 (530h23m19.752027711s) [seg_pos={0 0}, upload=465160]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:52:19.881197798 (530h23m19.752280833s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection
Dec 27 17:52:19.881206735 (530h23m19.75228978s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881335440 (530h23m19.752418475s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection
Dec 27 17:52:19.881353675 (530h23m19.75243672s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881588842 (530h23m19.752671907s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: context canceled; context canceled
Dec 27 17:52:19.881598421 (530h23m19.752681456s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881624350 (530h23m19.752707395s) [seg_pos={0 0}, upload=465160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:52:19.881756241 (530h23m19.752839276s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:52:19.881765429 (530h23m19.752848464s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:52:20.169754369 (530h23m20.040837404s) [upload=465160]: Done uploading segments...
Dec 27 17:52:20.169772945 (530h23m20.04085598s) [upload=465160]: Upload finished. err: <nil>
Duration: 11.172425818s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.............................................X..............]
duration: 
3.636625409s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:55:01 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.636654454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..............................................X.............]
duration: 
3.630919766s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:53:49 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.630953841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X..]
duration: 
6.013381996s
trace: 
https://jaeger.dev.storj.io/trace/7f227738e815068d
FAIL (2025-12-27 17:53:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
loading segments...
Dec 27 17:52:17.271669397 (530h23m17.142752442s) [upload=465160]: Got next segment. Inline: false
Dec 27 17:52:17.271700696 (530h23m17.142783741s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:52:17.271705495 (530h23m17.14278853s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:52:17.516501970 (530h23m17.387585005s) [seg_pos={0 0}, upload=465160]: Begin upload segment...
Dec 27 17:52:17.516548298 (530h23m17.387631333s) [seg_pos={0 0}, upload=465160]: Done begin upload segment.
Dec 27 17:52:17.516573536 (530h23m17.387656571s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.516647507 (530h23m17.387730552s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517273529 (530h23m17.388356564s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517671336 (530h23m17.388754381s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518066639 (530h23m17.389149684s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518520123 (530h23m17.389603158s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518955682 (530h23m17.390038717s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520080474 (530h23m17.391163509s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520388761 (530h23m17.391471796s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521286019 (530h23m17.392369054s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521940235 (530h23m17.39302327s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.522506884 (530h23m17.393589919s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.523750652 (530h23m17.394833687s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.524965336 (530h23m17.396048371s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.526192752 (530h23m17.397275798s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.528015083 (530h23m17.399098118s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:19.236153162 (530h23m19.107236197s) [upload=465160]: Next returned nil segment
Dec 27 17:52:19.236174633 (530h23m19.107257668s) [upload=465160]: Waiting for error group managing segments...
Dec 27 17:52:19.531582360 (530h23m19.402665405s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.531965430 (530h23m19.403048465s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.536525944 (530h23m19.407608989s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540017642 (530h23m19.411100687s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540394209 (530h23m19.411477244s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540740749 (530h23m19.411823784s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556124372 (530h23m19.427207407s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556473507 (530h23m19.427556552s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.559846770 (530h23m19.430929816s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.775170735 (530h23m19.64625378s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.793054708 (530h23m19.664137753s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.831761974 (530h23m19.702845009s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880923065 (530h23m19.7520061s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880944676 (530h23m19.752027711s) [seg_pos={0 0}, upload=465160]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:52:19.881197798 (530h23m19.752280833s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection
Dec 27 17:52:19.881206735 (530h23m19.75228978s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881335440 (530h23m19.752418475s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection
Dec 27 17:52:19.881353675 (530h23m19.75243672s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881588842 (530h23m19.752671907s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: context canceled; context canceled
Dec 27 17:52:19.881598421 (530h23m19.752681456s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881624350 (530h23m19.752707395s) [seg_pos={0 0}, upload=465160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:52:19.881756241 (530h23m19.752839276s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:52:19.881765429 (530h23m19.752848464s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:52:20.169754369 (530h23m20.040837404s) [upload=465160]: Done uploading segments...
Dec 27 17:52:20.169772945 (530h23m20.04085598s) [upload=465160]: Upload finished. err: <nil>
Duration: 6.013409029s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...............................................X............]
duration: 
3.573329944s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:53:18 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.573517172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [.X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X.]
duration: 
6.182658546s
trace: 
https://jaeger.dev.storj.io/trace/7f227738e815068d
FAIL (2025-12-27 17:52:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
loading segments...
Dec 27 17:52:17.271669397 (530h23m17.142752442s) [upload=465160]: Got next segment. Inline: false
Dec 27 17:52:17.271700696 (530h23m17.142783741s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:52:17.271705495 (530h23m17.14278853s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:52:17.516501970 (530h23m17.387585005s) [seg_pos={0 0}, upload=465160]: Begin upload segment...
Dec 27 17:52:17.516548298 (530h23m17.387631333s) [seg_pos={0 0}, upload=465160]: Done begin upload segment.
Dec 27 17:52:17.516573536 (530h23m17.387656571s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.516647507 (530h23m17.387730552s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517273529 (530h23m17.388356564s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517671336 (530h23m17.388754381s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518066639 (530h23m17.389149684s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518520123 (530h23m17.389603158s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518955682 (530h23m17.390038717s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520080474 (530h23m17.391163509s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520388761 (530h23m17.391471796s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521286019 (530h23m17.392369054s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521940235 (530h23m17.39302327s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.522506884 (530h23m17.393589919s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.523750652 (530h23m17.394833687s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.524965336 (530h23m17.396048371s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.526192752 (530h23m17.397275798s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.528015083 (530h23m17.399098118s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:19.236153162 (530h23m19.107236197s) [upload=465160]: Next returned nil segment
Dec 27 17:52:19.236174633 (530h23m19.107257668s) [upload=465160]: Waiting for error group managing segments...
Dec 27 17:52:19.531582360 (530h23m19.402665405s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.531965430 (530h23m19.403048465s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.536525944 (530h23m19.407608989s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540017642 (530h23m19.411100687s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540394209 (530h23m19.411477244s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540740749 (530h23m19.411823784s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556124372 (530h23m19.427207407s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556473507 (530h23m19.427556552s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.559846770 (530h23m19.430929816s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.775170735 (530h23m19.64625378s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.793054708 (530h23m19.664137753s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.831761974 (530h23m19.702845009s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880923065 (530h23m19.7520061s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880944676 (530h23m19.752027711s) [seg_pos={0 0}, upload=465160]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:52:19.881197798 (530h23m19.752280833s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection
Dec 27 17:52:19.881206735 (530h23m19.75228978s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881335440 (530h23m19.752418475s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection
Dec 27 17:52:19.881353675 (530h23m19.75243672s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881588842 (530h23m19.752671907s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: context canceled; context canceled
Dec 27 17:52:19.881598421 (530h23m19.752681456s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881624350 (530h23m19.752707395s) [seg_pos={0 0}, upload=465160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:52:19.881756241 (530h23m19.752839276s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:52:19.881765429 (530h23m19.752848464s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:52:20.169754369 (530h23m20.040837404s) [upload=465160]: Done uploading segments...
Dec 27 17:52:20.169772945 (530h23m20.04085598s) [upload=465160]: Upload finished. err: <nil>
Duration: 6.182673565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [................................................X...........]
duration: 
3.582357055s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:52:48 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.582384427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [..X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....X]
duration: 
30.004122227s
trace: 
https://jaeger.dev.storj.io/trace/7f227738e815068d
FAIL (2025-12-27 17:52:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
loading segments...
Dec 27 17:52:17.271669397 (530h23m17.142752442s) [upload=465160]: Got next segment. Inline: false
Dec 27 17:52:17.271700696 (530h23m17.142783741s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:52:17.271705495 (530h23m17.14278853s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:52:17.516501970 (530h23m17.387585005s) [seg_pos={0 0}, upload=465160]: Begin upload segment...
Dec 27 17:52:17.516548298 (530h23m17.387631333s) [seg_pos={0 0}, upload=465160]: Done begin upload segment.
Dec 27 17:52:17.516573536 (530h23m17.387656571s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.516647507 (530h23m17.387730552s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517273529 (530h23m17.388356564s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.517671336 (530h23m17.388754381s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518066639 (530h23m17.389149684s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518520123 (530h23m17.389603158s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.518955682 (530h23m17.390038717s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520080474 (530h23m17.391163509s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.520388761 (530h23m17.391471796s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521286019 (530h23m17.392369054s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.521940235 (530h23m17.39302327s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.522506884 (530h23m17.393589919s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.523750652 (530h23m17.394833687s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.524965336 (530h23m17.396048371s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.526192752 (530h23m17.397275798s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:17.528015083 (530h23m17.399098118s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Uploading piece...
Dec 27 17:52:19.236153162 (530h23m19.107236197s) [upload=465160]: Next returned nil segment
Dec 27 17:52:19.236174633 (530h23m19.107257668s) [upload=465160]: Waiting for error group managing segments...
Dec 27 17:52:19.531582360 (530h23m19.402665405s) [noise=true , address=145.79.227.17:4972   , piece_id=IGEVIOEZARSPPENXN5YN5C2DWGUDOTZWGJVEVV4CAGRE52ZOU7QA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.531965430 (530h23m19.403048465s) [noise=true , address=145.79.227.15:4876   , piece_id=FJ4RTV3WUM7X6V5CAOMAPPASFNCG5YTXIJNNUUTWOV6R2OC6QENQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.536525944 (530h23m19.407608989s) [noise=true , address=145.79.227.37:4567   , piece_id=WATUKYEK7S7CO24NWQJ4DKX7LA6GFDI3QKNYUJHTBBGLLJA6UM7A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540017642 (530h23m19.411100687s) [noise=true , address=145.79.227.48:4303   , piece_id=6V5GHQ4WMIYAHMU4S3JRPD3CFBOLHWT7QMAQOLVB4OLVTSRAKBUA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540394209 (530h23m19.411477244s) [noise=true , address=145.79.227.10:4403   , piece_id=JLZJAQUNCJJZSYL5TBHWSQE5ZORB5NTJUW3IZNP5KJUB3VUP5GEQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.540740749 (530h23m19.411823784s) [noise=true , address=145.79.227.42:4122   , piece_id=TUAMNCLE7HSJEDRYWZS5J63HO4KCQXSVFG6OQTEWNP2LCAUQXINQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556124372 (530h23m19.427207407s) [noise=true , address=145.79.227.20:4438   , piece_id=THKI63E3JK77XS3WCRVA5JNHT5LDI3L7VDTNMVC7GVJT3CX4TKLA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.556473507 (530h23m19.427556552s) [noise=true , address=145.79.227.14:4837   , piece_id=VTK7YXCMAPVQZD7SKVCCLSJC6TMA4MA3PFASLMB7ABBS4D5HCFGA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.559846770 (530h23m19.430929816s) [noise=true , address=145.79.227.36:4519   , piece_id=5Z6DFSPBEZC4KZJHY3Z3IXPVUL5VAJVVHWW4UEE2N6C4TFKISSDQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.775170735 (530h23m19.64625378s) [noise=true , address=77.90.31.14:4661     , piece_id=2CCN67VXQK6MKTVKAEANFUNRCDWJGLGNHIFL2IGYAS5IA4PUV6NA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.793054708 (530h23m19.664137753s) [noise=true , address=45.91.116.9:4263     , piece_id=OERBQKRMVETUGVR2SFGMKZD6BVAG2TK4SXHN2LU2OI5UW3UFCCAA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.831761974 (530h23m19.702845009s) [noise=true , address=45.91.116.5:4300     , piece_id=EUORJZL56QSPZS4Z72YEOJULL6ZP4LFQF6L6VOQMTLYNDS5OTD4A, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880923065 (530h23m19.7520061s) [noise=true , address=45.91.116.13:4153    , piece_id=UUVKFGJ4U4MBHGIV2YFJAQGJYCXHMB274MNO7XJXBNO4G63XTRCA, seg_pos={0 0}, upload=465160]: Done uploading piece. err: <nil>
Dec 27 17:52:19.880944676 (530h23m19.752027711s) [seg_pos={0 0}, upload=465160]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:52:19.881197798 (530h23m19.752280833s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50470->145.79.227.2:4114: use of closed network connection
Dec 27 17:52:19.881206735 (530h23m19.75228978s) [noise=true , address=145.79.227.2:4114    , piece_id=XOAJM4IZBZVZZ6VET4OG44T6QPMXGVVOHW2KWO6LZ7RGNJVDS4TQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881335440 (530h23m19.752418475s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51880->145.79.227.4:4186: use of closed network connection
Dec 27 17:52:19.881353675 (530h23m19.75243672s) [noise=true , address=145.79.227.4:4186    , piece_id=PJBFERQBEPSFBD5YTVW5OLXYLAH5RNYBL4U3TO564HG6AREGT7DQ, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881588842 (530h23m19.752671907s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: context canceled; context canceled
Dec 27 17:52:19.881598421 (530h23m19.752681456s) [noise=true , address=45.91.116.14:4190    , piece_id=56RD54YP4EJFQEAWYBQ4TCJQEZCG3HY4WJFLP3IB3AC2MU5QTL3Q, seg_pos={0 0}, upload=465160]: Long tail cancellation due to optimal threshold
Dec 27 17:52:19.881624350 (530h23m19.752707395s) [seg_pos={0 0}, upload=465160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:52:19.881756241 (530h23m19.752839276s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:52:19.881765429 (530h23m19.752848464s) [upload=465160]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:52:20.169754369 (530h23m20.040837404s) [upload=465160]: Done uploading segments...
Dec 27 17:52:20.169772945 (530h23m20.04085598s) [upload=465160]: Upload finished. err: <nil>
Duration: 30.004142725s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.................................................X..........]
duration: 
3.581716875s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:52:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.581853777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX....]
duration: 
5.907579379s
trace: 
https://jaeger.dev.storj.io/trace/209a2f7055728b6a
FAIL (2025-12-27 17:51:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ) context deadline exceeded
======LOGS======
) [upload=465095]: Uploading segments...
Dec 27 17:47:59.801444713 (530h18m59.672527778s) [upload=465095]: Got next segment. Inline: false
Dec 27 17:47:59.801487544 (530h18m59.672570579s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:47:59.801494036 (530h18m59.672577091s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:48:00.051992809 (530h18m59.923075854s) [seg_pos={0 0}, upload=465095]: Begin upload segment...
Dec 27 17:48:00.052054917 (530h18m59.923137962s) [seg_pos={0 0}, upload=465095]: Done begin upload segment.
Dec 27 17:48:00.052090174 (530h18m59.92317322s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052171410 (530h18m59.923254445s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052973076 (530h18m59.924056111s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053233021 (530h18m59.924316096s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053655425 (530h18m59.92473847s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054009008 (530h18m59.925092043s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054225181 (530h18m59.925308216s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054343886 (530h18m59.925426921s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054771981 (530h18m59.925855026s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054917879 (530h18m59.926000924s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055081260 (530h18m59.926164295s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055373738 (530h18m59.926456773s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055459981 (530h18m59.926543026s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055675632 (530h18m59.926758677s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055997865 (530h18m59.9270809s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.057884819 (530h18m59.928967864s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:01.768860351 (530h19m1.639943396s) [upload=465095]: Next returned nil segment
Dec 27 17:48:01.768880018 (530h19m1.639963063s) [upload=465095]: Waiting for error group managing segments...
Dec 27 17:48:02.043969867 (530h19m1.915052922s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.065296096 (530h19m1.936379141s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.068743420 (530h19m1.939826466s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070290666 (530h19m1.941373711s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070750221 (530h19m1.941833266s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.075848259 (530h19m1.946931294s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.079186515 (530h19m1.95026956s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.080881693 (530h19m1.951964738s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.089783353 (530h19m1.960866388s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.104536566 (530h19m1.975619611s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.289606171 (530h19m2.160689206s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.301165902 (530h19m2.172248947s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416915750 (530h19m2.287998825s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416958281 (530h19m2.288041346s) [seg_pos={0 0}, upload=465095]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:48:02.417235849 (530h19m2.288318904s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 17:48:02.417246190 (530h19m2.288329235s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417512726 (530h19m2.288595772s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection
Dec 27 17:48:02.417552212 (530h19m2.288635257s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417657953 (530h19m2.288741018s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection
Dec 27 17:48:02.417669515 (530h19m2.288752561s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417740791 (530h19m2.288823836s) [seg_pos={0 0}, upload=465095]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:48:02.417961391 (530h19m2.289044436s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:48:02.417971239 (530h19m2.289054274s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:48:02.707793531 (530h19m2.578876576s) [upload=465095]: Done uploading segments...
Dec 27 17:48:02.707812929 (530h19m2.578895964s) [upload=465095]: Upload finished. err: <nil>
Duration: 5.907604957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX...]
duration: 
24.22364311s
trace: 
https://jaeger.dev.storj.io/trace/209a2f7055728b6a
FAIL (2025-12-27 17:51:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ) context deadline exceeded
======LOGS======
) [upload=465095]: Uploading segments...
Dec 27 17:47:59.801444713 (530h18m59.672527778s) [upload=465095]: Got next segment. Inline: false
Dec 27 17:47:59.801487544 (530h18m59.672570579s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:47:59.801494036 (530h18m59.672577091s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:48:00.051992809 (530h18m59.923075854s) [seg_pos={0 0}, upload=465095]: Begin upload segment...
Dec 27 17:48:00.052054917 (530h18m59.923137962s) [seg_pos={0 0}, upload=465095]: Done begin upload segment.
Dec 27 17:48:00.052090174 (530h18m59.92317322s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052171410 (530h18m59.923254445s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052973076 (530h18m59.924056111s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053233021 (530h18m59.924316096s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053655425 (530h18m59.92473847s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054009008 (530h18m59.925092043s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054225181 (530h18m59.925308216s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054343886 (530h18m59.925426921s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054771981 (530h18m59.925855026s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054917879 (530h18m59.926000924s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055081260 (530h18m59.926164295s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055373738 (530h18m59.926456773s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055459981 (530h18m59.926543026s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055675632 (530h18m59.926758677s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055997865 (530h18m59.9270809s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.057884819 (530h18m59.928967864s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:01.768860351 (530h19m1.639943396s) [upload=465095]: Next returned nil segment
Dec 27 17:48:01.768880018 (530h19m1.639963063s) [upload=465095]: Waiting for error group managing segments...
Dec 27 17:48:02.043969867 (530h19m1.915052922s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.065296096 (530h19m1.936379141s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.068743420 (530h19m1.939826466s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070290666 (530h19m1.941373711s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070750221 (530h19m1.941833266s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.075848259 (530h19m1.946931294s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.079186515 (530h19m1.95026956s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.080881693 (530h19m1.951964738s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.089783353 (530h19m1.960866388s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.104536566 (530h19m1.975619611s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.289606171 (530h19m2.160689206s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.301165902 (530h19m2.172248947s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416915750 (530h19m2.287998825s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416958281 (530h19m2.288041346s) [seg_pos={0 0}, upload=465095]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:48:02.417235849 (530h19m2.288318904s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 17:48:02.417246190 (530h19m2.288329235s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417512726 (530h19m2.288595772s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection
Dec 27 17:48:02.417552212 (530h19m2.288635257s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417657953 (530h19m2.288741018s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection
Dec 27 17:48:02.417669515 (530h19m2.288752561s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417740791 (530h19m2.288823836s) [seg_pos={0 0}, upload=465095]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:48:02.417961391 (530h19m2.289044436s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:48:02.417971239 (530h19m2.289054274s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:48:02.707793531 (530h19m2.578876576s) [upload=465095]: Done uploading segments...
Dec 27 17:48:02.707812929 (530h19m2.578895964s) [upload=465095]: Upload finished. err: <nil>
Duration: 24.223672544s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..................................................X.........]
duration: 
3.576870998s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:51:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.577024941s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX..]
duration: 
5.888672175s
trace: 
https://jaeger.dev.storj.io/trace/209a2f7055728b6a
FAIL (2025-12-27 17:50:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ) context deadline exceeded
======LOGS======
) [upload=465095]: Uploading segments...
Dec 27 17:47:59.801444713 (530h18m59.672527778s) [upload=465095]: Got next segment. Inline: false
Dec 27 17:47:59.801487544 (530h18m59.672570579s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:47:59.801494036 (530h18m59.672577091s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:48:00.051992809 (530h18m59.923075854s) [seg_pos={0 0}, upload=465095]: Begin upload segment...
Dec 27 17:48:00.052054917 (530h18m59.923137962s) [seg_pos={0 0}, upload=465095]: Done begin upload segment.
Dec 27 17:48:00.052090174 (530h18m59.92317322s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052171410 (530h18m59.923254445s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052973076 (530h18m59.924056111s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053233021 (530h18m59.924316096s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053655425 (530h18m59.92473847s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054009008 (530h18m59.925092043s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054225181 (530h18m59.925308216s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054343886 (530h18m59.925426921s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054771981 (530h18m59.925855026s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054917879 (530h18m59.926000924s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055081260 (530h18m59.926164295s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055373738 (530h18m59.926456773s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055459981 (530h18m59.926543026s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055675632 (530h18m59.926758677s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055997865 (530h18m59.9270809s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.057884819 (530h18m59.928967864s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:01.768860351 (530h19m1.639943396s) [upload=465095]: Next returned nil segment
Dec 27 17:48:01.768880018 (530h19m1.639963063s) [upload=465095]: Waiting for error group managing segments...
Dec 27 17:48:02.043969867 (530h19m1.915052922s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.065296096 (530h19m1.936379141s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.068743420 (530h19m1.939826466s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070290666 (530h19m1.941373711s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070750221 (530h19m1.941833266s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.075848259 (530h19m1.946931294s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.079186515 (530h19m1.95026956s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.080881693 (530h19m1.951964738s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.089783353 (530h19m1.960866388s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.104536566 (530h19m1.975619611s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.289606171 (530h19m2.160689206s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.301165902 (530h19m2.172248947s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416915750 (530h19m2.287998825s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416958281 (530h19m2.288041346s) [seg_pos={0 0}, upload=465095]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:48:02.417235849 (530h19m2.288318904s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 17:48:02.417246190 (530h19m2.288329235s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417512726 (530h19m2.288595772s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection
Dec 27 17:48:02.417552212 (530h19m2.288635257s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417657953 (530h19m2.288741018s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection
Dec 27 17:48:02.417669515 (530h19m2.288752561s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417740791 (530h19m2.288823836s) [seg_pos={0 0}, upload=465095]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:48:02.417961391 (530h19m2.289044436s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:48:02.417971239 (530h19m2.289054274s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:48:02.707793531 (530h19m2.578876576s) [upload=465095]: Done uploading segments...
Dec 27 17:48:02.707812929 (530h19m2.578895964s) [upload=465095]: Upload finished. err: <nil>
Duration: 5.888702403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...................................................X........]
duration: 
3.651696931s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:49:56 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.651737688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX.]
duration: 
6.012965692s
trace: 
https://jaeger.dev.storj.io/trace/209a2f7055728b6a
FAIL (2025-12-27 17:49:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ) context deadline exceeded
======LOGS======
) [upload=465095]: Uploading segments...
Dec 27 17:47:59.801444713 (530h18m59.672527778s) [upload=465095]: Got next segment. Inline: false
Dec 27 17:47:59.801487544 (530h18m59.672570579s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:47:59.801494036 (530h18m59.672577091s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:48:00.051992809 (530h18m59.923075854s) [seg_pos={0 0}, upload=465095]: Begin upload segment...
Dec 27 17:48:00.052054917 (530h18m59.923137962s) [seg_pos={0 0}, upload=465095]: Done begin upload segment.
Dec 27 17:48:00.052090174 (530h18m59.92317322s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052171410 (530h18m59.923254445s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052973076 (530h18m59.924056111s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053233021 (530h18m59.924316096s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053655425 (530h18m59.92473847s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054009008 (530h18m59.925092043s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054225181 (530h18m59.925308216s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054343886 (530h18m59.925426921s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054771981 (530h18m59.925855026s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054917879 (530h18m59.926000924s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055081260 (530h18m59.926164295s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055373738 (530h18m59.926456773s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055459981 (530h18m59.926543026s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055675632 (530h18m59.926758677s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055997865 (530h18m59.9270809s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.057884819 (530h18m59.928967864s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:01.768860351 (530h19m1.639943396s) [upload=465095]: Next returned nil segment
Dec 27 17:48:01.768880018 (530h19m1.639963063s) [upload=465095]: Waiting for error group managing segments...
Dec 27 17:48:02.043969867 (530h19m1.915052922s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.065296096 (530h19m1.936379141s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.068743420 (530h19m1.939826466s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070290666 (530h19m1.941373711s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070750221 (530h19m1.941833266s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.075848259 (530h19m1.946931294s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.079186515 (530h19m1.95026956s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.080881693 (530h19m1.951964738s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.089783353 (530h19m1.960866388s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.104536566 (530h19m1.975619611s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.289606171 (530h19m2.160689206s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.301165902 (530h19m2.172248947s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416915750 (530h19m2.287998825s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416958281 (530h19m2.288041346s) [seg_pos={0 0}, upload=465095]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:48:02.417235849 (530h19m2.288318904s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 17:48:02.417246190 (530h19m2.288329235s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417512726 (530h19m2.288595772s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection
Dec 27 17:48:02.417552212 (530h19m2.288635257s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417657953 (530h19m2.288741018s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection
Dec 27 17:48:02.417669515 (530h19m2.288752561s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417740791 (530h19m2.288823836s) [seg_pos={0 0}, upload=465095]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:48:02.417961391 (530h19m2.289044436s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:48:02.417971239 (530h19m2.289054274s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:48:02.707793531 (530h19m2.578876576s) [upload=465095]: Done uploading segments...
Dec 27 17:48:02.707812929 (530h19m2.578895964s) [upload=465095]: Upload finished. err: <nil>
Duration: 6.012987453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [....................................................X.......]
duration: 
3.699654149s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:48:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.699681901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..XX]
duration: 
30.002959291s
trace: 
https://jaeger.dev.storj.io/trace/209a2f7055728b6a
FAIL (2025-12-27 17:48:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ) context deadline exceeded
======LOGS======
) [upload=465095]: Uploading segments...
Dec 27 17:47:59.801444713 (530h18m59.672527778s) [upload=465095]: Got next segment. Inline: false
Dec 27 17:47:59.801487544 (530h18m59.672570579s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:47:59.801494036 (530h18m59.672577091s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:48:00.051992809 (530h18m59.923075854s) [seg_pos={0 0}, upload=465095]: Begin upload segment...
Dec 27 17:48:00.052054917 (530h18m59.923137962s) [seg_pos={0 0}, upload=465095]: Done begin upload segment.
Dec 27 17:48:00.052090174 (530h18m59.92317322s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052171410 (530h18m59.923254445s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.052973076 (530h18m59.924056111s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053233021 (530h18m59.924316096s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.053655425 (530h18m59.92473847s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054009008 (530h18m59.925092043s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054225181 (530h18m59.925308216s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054343886 (530h18m59.925426921s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054771981 (530h18m59.925855026s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.054917879 (530h18m59.926000924s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055081260 (530h18m59.926164295s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055373738 (530h18m59.926456773s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055459981 (530h18m59.926543026s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055675632 (530h18m59.926758677s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.055997865 (530h18m59.9270809s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:00.057884819 (530h18m59.928967864s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Uploading piece...
Dec 27 17:48:01.768860351 (530h19m1.639943396s) [upload=465095]: Next returned nil segment
Dec 27 17:48:01.768880018 (530h19m1.639963063s) [upload=465095]: Waiting for error group managing segments...
Dec 27 17:48:02.043969867 (530h19m1.915052922s) [noise=true , address=145.79.227.15:4876   , piece_id=FDYJEXO4BIGC6N7NTRJPU52SHWH7GIQ4GQFUUM7LEG7MJZTVHIRQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.065296096 (530h19m1.936379141s) [noise=true , address=145.79.227.23:4545   , piece_id=QIUOEV3DYCRQH7CTD7WBTX4E56EQS5AGTYONDIEPZWRNMUU4SFEQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.068743420 (530h19m1.939826466s) [noise=true , address=145.79.227.48:4302   , piece_id=PQI6I5QNDTE7J3B32I5KJL7QRYQJ7VC4GNSAMST3WWHNSZ7OYIUA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070290666 (530h19m1.941373711s) [noise=true , address=145.79.227.39:4014   , piece_id=DDL4NSOOKRHGAL4S6N7TKJC6GP6XP2J5K43BCSQC65JGUYVSD6HQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.070750221 (530h19m1.941833266s) [noise=true , address=145.79.227.36:4519   , piece_id=TCAZD3VZU35RG5MUR3WD2C6OR4JTWD6KFV63XBD6G5QKSDLNEFGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.075848259 (530h19m1.946931294s) [noise=true , address=145.79.227.51:4411   , piece_id=7IN4CRPXCF3OK5ZYSAD3CHXXVWFO3WSGRBSC34LRWZOD5ZTWBW6A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.079186515 (530h19m1.95026956s) [noise=true , address=145.79.227.9:4365    , piece_id=QLLNM4DPCUAOXQPHHDAZZU7N4A42F6GKJV3ZHESMIY6E4ZLFP7EQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.080881693 (530h19m1.951964738s) [noise=true , address=145.79.227.3:4151    , piece_id=AAQVJQ6RGIDTZGHMHGNHO6N7OVQOQZSZT3TAAWKFPZYKRJ6666AA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.089783353 (530h19m1.960866388s) [noise=true , address=145.79.227.38:4977   , piece_id=LVFMKZ4US5AGOCNLUCGI3YTY6RTJUF2HLDT7QIXSTBMMOU34FQSA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.104536566 (530h19m1.975619611s) [noise=true , address=145.79.227.42:4121   , piece_id=BAZRG3YMJUJR2RHLHZS7FYZPCAOW5HHLDVEFLNZTPX3FI5GSBC2A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.289606171 (530h19m2.160689206s) [noise=true , address=45.91.116.8:4045     , piece_id=CXS3RXGYN5CGMAHWOYETILF5AFQIC5I74QOEJJKGINMRIFR4ECGQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.301165902 (530h19m2.172248947s) [noise=true , address=45.91.116.13:4153    , piece_id=MH76M2KVWKU2INMH7FHQQDKIOPP34PZ6N2IADXOZ3GBLHDPB54IQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416915750 (530h19m2.287998825s) [noise=true , address=45.91.116.11:4117    , piece_id=JJ5XWLVOW3DTG3GLCMDA6GN5MJDFMPKVEB2M7NDQ3TO6PZHJO5WQ, seg_pos={0 0}, upload=465095]: Done uploading piece. err: <nil>
Dec 27 17:48:02.416958281 (530h19m2.288041346s) [seg_pos={0 0}, upload=465095]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:48:02.417235849 (530h19m2.288318904s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 17:48:02.417246190 (530h19m2.288329235s) [noise=true , address=77.90.31.10:4518     , piece_id=LLM4MVEEGPNMQ5D6C2YBESCATHUPEVC5SR7NAZMMZT3RV57NIZ7A, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417512726 (530h19m2.288595772s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37692->45.91.116.9:4263: use of closed network connection
Dec 27 17:48:02.417552212 (530h19m2.288635257s) [noise=true , address=45.91.116.9:4263     , piece_id=DJQJOY5TRNYZQQNTXESR5NK4TBPKJ6ZEJHQ4EUPF6P7S5FD7A4NA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417657953 (530h19m2.288741018s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50136->45.91.116.6:4976: use of closed network connection
Dec 27 17:48:02.417669515 (530h19m2.288752561s) [noise=true , address=45.91.116.6:4976     , piece_id=2LTM5FH6IWY3JEEBQFAL5T7ZB473S7ED7QQRWO3SNFGXDAV4EQYA, seg_pos={0 0}, upload=465095]: Long tail cancellation due to optimal threshold
Dec 27 17:48:02.417740791 (530h19m2.288823836s) [seg_pos={0 0}, upload=465095]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:48:02.417961391 (530h19m2.289044436s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:48:02.417971239 (530h19m2.289054274s) [upload=465095]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:48:02.707793531 (530h19m2.578876576s) [upload=465095]: Done uploading segments...
Dec 27 17:48:02.707812929 (530h19m2.578895964s) [upload=465095]: Upload finished. err: <nil>
Duration: 30.003066796s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.....................................................X......]
duration: 
3.534356029s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:48:28 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.53438782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......................................................X.....]
duration: 
3.60165931s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:48:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.601683676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [..X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..X]
duration: 
30.003006592s
trace: 
https://jaeger.dev.storj.io/trace/7221dc91da135a97
FAIL (2025-12-27 17:47:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: M6EB5TZONHWXLWK43FFBKAQ7AQ3LEUNAZFBWJKWO5AXHSNWKMDIA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: FTOM6IILCAREHMO75QZSPCZZI3K3IJ6KOU2JONJLBGN5N4EV4MUA) context deadline exceeded
======LOGS======
Dec 27 17:46:59.997791986 (530h17m59.868875031s) [upload=465076]: Starting upload
Dec 27 17:46:59.997895313 (530h17m59.868978358s) [upload=465076]: Done starting upload
Dec 27 17:46:59.998122856 (530h17m59.869205901s) [upload=465076]: Uploading segments...
Dec 27 17:46:59.998193942 (530h17m59.869276987s) [upload=465076]: Got next segment. Inline: false
Dec 27 17:46:59.998237104 (530h17m59.869320139s) [upload=465076]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:46:59.998245409 (530h17m59.869328455s) [upload=465076]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:47:00.244530147 (530h18m0.115613182s) [seg_pos={0 0}, upload=465076]: Begin upload segment...
Dec 27 17:47:00.244588218 (530h18m0.115671273s) [seg_pos={0 0}, upload=465076]: Done begin upload segment.
Dec 27 17:47:00.244618546 (530h18m0.115701581s) [noise=true , address=77.90.31.13:4627     , piece_id=Q2Y5AM77ANLZCRGM5FCYBGICTCJZRXEZXEN24DDQU7QP4GGMF4PQ, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.244770134 (530h18m0.115853179s) [noise=true , address=145.79.227.48:4301   , piece_id=QJ7BCOYEXVAG2AKQFDR6EBJRDCIVH5FVY2UIW72GTELVSUT7DZLA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.245477873 (530h18m0.116560918s) [noise=true , address=77.90.31.16:4735     , piece_id=O7UBNGNW3EPGGO7I3CVZRPL4LYRTA34SPBRV224KTSJO64DHJ5EQ, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.246125004 (530h18m0.117208049s) [noise=true , address=45.91.116.11:4117    , piece_id=BWRMF23F3CO4A6JYC4QKRNVUL7V3NL45OM2PFFAFJXANQZW3Y2AA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.246668079 (530h18m0.117751114s) [noise=true , address=145.79.227.16:4910   , piece_id=EPRXJTYIDLWV5AJ7N7U474LFDQCKQCVK4YQNQITBSZ3A4GTY662A, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.247152000 (530h18m0.118235045s) [noise=true , address=145.79.227.37:4565   , piece_id=WVIECCEKXC6YLN2K42WZWR3QKRPF4WM7UYNS44FI4DLQDADAH23A, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.247608740 (530h18m0.118691785s) [noise=true , address=45.91.116.14:4192    , piece_id=FMNGDTOAA3RLDDAKJLRMI2Z4GOVL444IZEOCQMDXEE2DKCLUZ3DA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.248569129 (530h18m0.119652174s) [noise=true , address=145.79.227.52:4447   , piece_id=SDAZ5MDZ2AMMLJCT56H5WEDO2SRZGBJOCIDEHALFMPKWK4IBJI3A, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.249127993 (530h18m0.120211048s) [noise=true , address=45.91.116.5:4297     , piece_id=FTOM6IILCAREHMO75QZSPCZZI3K3IJ6KOU2JONJLBGN5N4EV4MUA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.249686607 (530h18m0.120769642s) [noise=true , address=145.79.227.2:4114    , piece_id=ETP3PEQA7FTT35PUK47UW7QZNTYP46JSLMXY3VB2HKKFQQOVM47Q, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.251093205 (530h18m0.12217625s) [noise=true , address=145.79.227.45:4229   , piece_id=YB2TRPAOOIZQT2D5UQ7D5J3WM5ELXPLPZZHJP3WPMERXBKPUHTIQ, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.253483897 (530h18m0.124566932s) [noise=true , address=45.91.116.7:4009     , piece_id=GDCHMMPPUL3ZSNRCJ4KYUTZQFTVSSUQIIP4TPURTG7SPHA5X2XCA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.256101361 (530h18m0.127184406s) [noise=true , address=145.79.227.23:4546   , piece_id=UVBMVGIQ2BNR5RZEVBA5G735FA3RVPXEXCZTBOYX2PT4DTTVWK5A, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.257720003 (530h18m0.128803038s) [noise=true , address=77.90.31.21:4877     , piece_id=M6EB5TZONHWXLWK43FFBKAQ7AQ3LEUNAZFBWJKWO5AXHSNWKMDIA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.258342879 (530h18m0.129425924s) [noise=true , address=45.91.116.6:4975     , piece_id=ZDRPW6WXVNMCE4Z4I4EOIKU3YZCHBIPWIMEX4QYYOTPZPRW5KJAQ, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:00.259117094 (530h18m0.130200139s) [noise=true , address=145.79.227.12:4767   , piece_id=QB7MMWEUPJ6DZS5P2ZXQ5ZARR4MNUMZBU6XAJC6GWBCZMYB5JQEA, seg_pos={0 0}, upload=465076]: Uploading piece...
Dec 27 17:47:01.966598622 (530h18m1.837681677s) [upload=465076]: Next returned nil segment
Dec 27 17:47:01.966661162 (530h18m1.837744247s) [upload=465076]: Waiting for error group managing segments...
Dec 27 17:47:02.261291038 (530h18m2.132374083s) [noise=true , address=145.79.227.12:4767   , piece_id=QB7MMWEUPJ6DZS5P2ZXQ5ZARR4MNUMZBU6XAJC6GWBCZMYB5JQEA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.264639273 (530h18m2.135722319s) [noise=true , address=145.79.227.37:4565   , piece_id=WVIECCEKXC6YLN2K42WZWR3QKRPF4WM7UYNS44FI4DLQDADAH23A, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.270846483 (530h18m2.141929528s) [noise=true , address=145.79.227.52:4447   , piece_id=SDAZ5MDZ2AMMLJCT56H5WEDO2SRZGBJOCIDEHALFMPKWK4IBJI3A, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.276047557 (530h18m2.147130602s) [noise=true , address=145.79.227.48:4301   , piece_id=QJ7BCOYEXVAG2AKQFDR6EBJRDCIVH5FVY2UIW72GTELVSUT7DZLA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.276165070 (530h18m2.147248116s) [noise=true , address=145.79.227.23:4546   , piece_id=UVBMVGIQ2BNR5RZEVBA5G735FA3RVPXEXCZTBOYX2PT4DTTVWK5A, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.277156329 (530h18m2.148239374s) [noise=true , address=145.79.227.16:4910   , piece_id=EPRXJTYIDLWV5AJ7N7U474LFDQCKQCVK4YQNQITBSZ3A4GTY662A, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.291557410 (530h18m2.162640465s) [noise=true , address=145.79.227.45:4229   , piece_id=YB2TRPAOOIZQT2D5UQ7D5J3WM5ELXPLPZZHJP3WPMERXBKPUHTIQ, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.483952383 (530h18m2.355035428s) [noise=true , address=45.91.116.5:4297     , piece_id=FTOM6IILCAREHMO75QZSPCZZI3K3IJ6KOU2JONJLBGN5N4EV4MUA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.491321746 (530h18m2.362404781s) [noise=true , address=77.90.31.16:4735     , piece_id=O7UBNGNW3EPGGO7I3CVZRPL4LYRTA34SPBRV224KTSJO64DHJ5EQ, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.529883915 (530h18m2.40096696s) [noise=true , address=45.91.116.11:4117    , piece_id=BWRMF23F3CO4A6JYC4QKRNVUL7V3NL45OM2PFFAFJXANQZW3Y2AA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.538118715 (530h18m2.40920176s) [noise=true , address=77.90.31.13:4627     , piece_id=Q2Y5AM77ANLZCRGM5FCYBGICTCJZRXEZXEN24DDQU7QP4GGMF4PQ, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.553679927 (530h18m2.424762972s) [noise=true , address=77.90.31.21:4877     , piece_id=M6EB5TZONHWXLWK43FFBKAQ7AQ3LEUNAZFBWJKWO5AXHSNWKMDIA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.600622754 (530h18m2.471705799s) [noise=true , address=45.91.116.6:4975     , piece_id=ZDRPW6WXVNMCE4Z4I4EOIKU3YZCHBIPWIMEX4QYYOTPZPRW5KJAQ, seg_pos={0 0}, upload=465076]: Done uploading piece. err: <nil>
Dec 27 17:47:02.600650847 (530h18m2.471733882s) [seg_pos={0 0}, upload=465076]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:47:02.600912446 (530h18m2.471995501s) [noise=true , address=45.91.116.14:4192    , piece_id=FMNGDTOAA3RLDDAKJLRMI2Z4GOVL444IZEOCQMDXEE2DKCLUZ3DA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 27 17:47:02.600921683 (530h18m2.472004728s) [noise=true , address=45.91.116.14:4192    , piece_id=FMNGDTOAA3RLDDAKJLRMI2Z4GOVL444IZEOCQMDXEE2DKCLUZ3DA, seg_pos={0 0}, upload=465076]: Long tail cancellation due to optimal threshold
Dec 27 17:47:02.601155298 (530h18m2.472238343s) [noise=true , address=145.79.227.2:4114    , piece_id=ETP3PEQA7FTT35PUK47UW7QZNTYP46JSLMXY3VB2HKKFQQOVM47Q, seg_pos={0 0}, upload=465076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:53442->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:53442->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53442->145.79.227.2:4114: use of closed network connection
Dec 27 17:47:02.601173723 (530h18m2.472256768s) [noise=true , address=145.79.227.2:4114    , piece_id=ETP3PEQA7FTT35PUK47UW7QZNTYP46JSLMXY3VB2HKKFQQOVM47Q, seg_pos={0 0}, upload=465076]: Long tail cancellation due to optimal threshold
Dec 27 17:47:02.601284504 (530h18m2.472367539s) [noise=true , address=45.91.116.7:4009     , piece_id=GDCHMMPPUL3ZSNRCJ4KYUTZQFTVSSUQIIP4TPURTG7SPHA5X2XCA, seg_pos={0 0}, upload=465076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 17:47:02.601295765 (530h18m2.472378811s) [noise=true , address=45.91.116.7:4009     , piece_id=GDCHMMPPUL3ZSNRCJ4KYUTZQFTVSSUQIIP4TPURTG7SPHA5X2XCA, seg_pos={0 0}, upload=465076]: Long tail cancellation due to optimal threshold
Dec 27 17:47:02.601409672 (530h18m2.472492747s) [seg_pos={0 0}, upload=465076]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:47:02.601794144 (530h18m2.472877179s) [upload=465076]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:47:02.601845723 (530h18m2.472928768s) [upload=465076]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:47:02.898947906 (530h18m2.770030951s) [upload=465076]: Done uploading segments...
Dec 27 17:47:02.899001138 (530h18m2.770084173s) [upload=465076]: Upload finished. err: <nil>
Duration: 30.003028084s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......................................................X....]
duration: 
3.553873968s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:47:14 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.553899107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [...X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX..]
duration: 
5.927445908s
trace: 
https://jaeger.dev.storj.io/trace/76cf65e4902332aa
FAIL (2025-12-27 17:46:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ) context deadline exceeded
======LOGS======
7:44:21.562113588 (530h15m21.433196623s) [upload=465039]: Starting upload
Dec 27 17:44:21.562194081 (530h15m21.433277126s) [upload=465039]: Done starting upload
Dec 27 17:44:21.562498761 (530h15m21.433581796s) [upload=465039]: Uploading segments...
Dec 27 17:44:21.562561200 (530h15m21.433644235s) [upload=465039]: Got next segment. Inline: false
Dec 27 17:44:21.562606506 (530h15m21.433689541s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:44:21.562612497 (530h15m21.433695542s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:44:21.809019469 (530h15m21.680102514s) [seg_pos={0 0}, upload=465039]: Begin upload segment...
Dec 27 17:44:21.809080986 (530h15m21.680164021s) [seg_pos={0 0}, upload=465039]: Done begin upload segment.
Dec 27 17:44:21.809111674 (530h15m21.680194709s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811298939 (530h15m21.682381984s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811881739 (530h15m21.682964774s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812289525 (530h15m21.68337257s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812668777 (530h15m21.683751822s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813043381 (530h15m21.684126416s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813410320 (530h15m21.684493365s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813782207 (530h15m21.684865242s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814318119 (530h15m21.685401164s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814736645 (530h15m21.68581968s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815208022 (530h15m21.686291068s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815634946 (530h15m21.686717991s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816114318 (530h15m21.687197353s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816750931 (530h15m21.687833966s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.817385409 (530h15m21.688468474s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.819636095 (530h15m21.69071914s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:23.539094660 (530h15m23.410177705s) [upload=465039]: Next returned nil segment
Dec 27 17:44:23.539116852 (530h15m23.410199897s) [upload=465039]: Waiting for error group managing segments...
Dec 27 17:44:23.822799252 (530h15m23.693882297s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.826893980 (530h15m23.697977015s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.828568979 (530h15m23.699652014s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.833374249 (530h15m23.704457284s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.839553706 (530h15m23.710636751s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.848655958 (530h15m23.719738993s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.855454203 (530h15m23.726537238s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.085651679 (530h15m23.956734724s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.116276664 (530h15m23.987359709s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.143078408 (530h15m24.014161453s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.161306428 (530h15m24.032389473s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.835346707 (530h15m24.706429742s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841033896 (530h15m24.712116961s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841058493 (530h15m24.712141528s) [seg_pos={0 0}, upload=465039]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:44:24.841288642 (530h15m24.712371717s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841298380 (530h15m24.712381425s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841476038 (530h15m24.712559093s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection
Dec 27 17:44:24.841485446 (530h15m24.712568491s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841682682 (530h15m24.712765727s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841692280 (530h15m24.712775315s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841713280 (530h15m24.712796315s) [seg_pos={0 0}, upload=465039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:44:24.841851554 (530h15m24.712934599s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:44:24.841859619 (530h15m24.712942664s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:44:25.141975962 (530h15m25.013059007s) [upload=465039]: Done uploading segments...
Dec 27 17:44:25.141997412 (530h15m25.013080447s) [upload=465039]: Upload finished. err: <nil>
Duration: 5.927476346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........................................................X...]
duration: 
3.501139332s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:46:09 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.501168017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................................................X..]
duration: 
3.57838446s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:45:46 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.578409056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX.]
duration: 
5.996528283s
trace: 
https://jaeger.dev.storj.io/trace/76cf65e4902332aa
FAIL (2025-12-27 17:45:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ) context deadline exceeded
======LOGS======
7:44:21.562113588 (530h15m21.433196623s) [upload=465039]: Starting upload
Dec 27 17:44:21.562194081 (530h15m21.433277126s) [upload=465039]: Done starting upload
Dec 27 17:44:21.562498761 (530h15m21.433581796s) [upload=465039]: Uploading segments...
Dec 27 17:44:21.562561200 (530h15m21.433644235s) [upload=465039]: Got next segment. Inline: false
Dec 27 17:44:21.562606506 (530h15m21.433689541s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:44:21.562612497 (530h15m21.433695542s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:44:21.809019469 (530h15m21.680102514s) [seg_pos={0 0}, upload=465039]: Begin upload segment...
Dec 27 17:44:21.809080986 (530h15m21.680164021s) [seg_pos={0 0}, upload=465039]: Done begin upload segment.
Dec 27 17:44:21.809111674 (530h15m21.680194709s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811298939 (530h15m21.682381984s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811881739 (530h15m21.682964774s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812289525 (530h15m21.68337257s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812668777 (530h15m21.683751822s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813043381 (530h15m21.684126416s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813410320 (530h15m21.684493365s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813782207 (530h15m21.684865242s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814318119 (530h15m21.685401164s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814736645 (530h15m21.68581968s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815208022 (530h15m21.686291068s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815634946 (530h15m21.686717991s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816114318 (530h15m21.687197353s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816750931 (530h15m21.687833966s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.817385409 (530h15m21.688468474s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.819636095 (530h15m21.69071914s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:23.539094660 (530h15m23.410177705s) [upload=465039]: Next returned nil segment
Dec 27 17:44:23.539116852 (530h15m23.410199897s) [upload=465039]: Waiting for error group managing segments...
Dec 27 17:44:23.822799252 (530h15m23.693882297s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.826893980 (530h15m23.697977015s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.828568979 (530h15m23.699652014s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.833374249 (530h15m23.704457284s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.839553706 (530h15m23.710636751s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.848655958 (530h15m23.719738993s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.855454203 (530h15m23.726537238s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.085651679 (530h15m23.956734724s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.116276664 (530h15m23.987359709s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.143078408 (530h15m24.014161453s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.161306428 (530h15m24.032389473s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.835346707 (530h15m24.706429742s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841033896 (530h15m24.712116961s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841058493 (530h15m24.712141528s) [seg_pos={0 0}, upload=465039]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:44:24.841288642 (530h15m24.712371717s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841298380 (530h15m24.712381425s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841476038 (530h15m24.712559093s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection
Dec 27 17:44:24.841485446 (530h15m24.712568491s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841682682 (530h15m24.712765727s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841692280 (530h15m24.712775315s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841713280 (530h15m24.712796315s) [seg_pos={0 0}, upload=465039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:44:24.841851554 (530h15m24.712934599s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:44:24.841859619 (530h15m24.712942664s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:44:25.141975962 (530h15m25.013059007s) [upload=465039]: Done uploading segments...
Dec 27 17:44:25.141997412 (530h15m25.013080447s) [upload=465039]: Upload finished. err: <nil>
Duration: 5.996554022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XXX]
duration: 
30.002277182s
trace: 
https://jaeger.dev.storj.io/trace/76cf65e4902332aa
FAIL (2025-12-27 17:44:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ) context deadline exceeded
======LOGS======
7:44:21.562113588 (530h15m21.433196623s) [upload=465039]: Starting upload
Dec 27 17:44:21.562194081 (530h15m21.433277126s) [upload=465039]: Done starting upload
Dec 27 17:44:21.562498761 (530h15m21.433581796s) [upload=465039]: Uploading segments...
Dec 27 17:44:21.562561200 (530h15m21.433644235s) [upload=465039]: Got next segment. Inline: false
Dec 27 17:44:21.562606506 (530h15m21.433689541s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:44:21.562612497 (530h15m21.433695542s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:44:21.809019469 (530h15m21.680102514s) [seg_pos={0 0}, upload=465039]: Begin upload segment...
Dec 27 17:44:21.809080986 (530h15m21.680164021s) [seg_pos={0 0}, upload=465039]: Done begin upload segment.
Dec 27 17:44:21.809111674 (530h15m21.680194709s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811298939 (530h15m21.682381984s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.811881739 (530h15m21.682964774s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812289525 (530h15m21.68337257s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.812668777 (530h15m21.683751822s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813043381 (530h15m21.684126416s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813410320 (530h15m21.684493365s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.813782207 (530h15m21.684865242s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814318119 (530h15m21.685401164s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.814736645 (530h15m21.68581968s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815208022 (530h15m21.686291068s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.815634946 (530h15m21.686717991s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816114318 (530h15m21.687197353s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.816750931 (530h15m21.687833966s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.817385409 (530h15m21.688468474s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:21.819636095 (530h15m21.69071914s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Uploading piece...
Dec 27 17:44:23.539094660 (530h15m23.410177705s) [upload=465039]: Next returned nil segment
Dec 27 17:44:23.539116852 (530h15m23.410199897s) [upload=465039]: Waiting for error group managing segments...
Dec 27 17:44:23.822799252 (530h15m23.693882297s) [noise=true , address=145.79.227.15:4876   , piece_id=2WVYVIU3RPZ4ZD7WWIXFBYQ4F3GAUZ4UTZSTG5NCT5PONQDA4YOA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.826893980 (530h15m23.697977015s) [noise=true , address=145.79.227.37:4566   , piece_id=H2YA2X3NORWHFCQOJ3X3P2WRNUIY7STG2EXR2QNXDCDCORWRD4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.828568979 (530h15m23.699652014s) [noise=true , address=145.79.227.38:4977   , piece_id=MS7QRLGBKEUYNWGOEPKJSJACFBSQGOZ6PB2UI7KUZTBCXHGVELJA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.833374249 (530h15m23.704457284s) [noise=true , address=145.79.227.2:4113    , piece_id=DLGO6HZZMP457JDXKPCM2ZK67ZSZMYGROYEXVXGCW6FKHBOMICDQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.839553706 (530h15m23.710636751s) [noise=true , address=145.79.227.52:4447   , piece_id=WT4BLYS5Y6KUSJUGV3PB6CT64FCQ5QY3VGBOSGMXAG3TWD3RECHA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.848655958 (530h15m23.719738993s) [noise=true , address=145.79.227.11:4729   , piece_id=4FAJZNDDLPE6T35JDHNZ4MGOTJRP2SOBA5HLJHPFWNTX5VEOO4RQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:23.855454203 (530h15m23.726537238s) [noise=true , address=145.79.227.32:4836   , piece_id=ESDB7WP23WZOF5ZSSA52UMAIVNLGH4ZSAGAI6IJIJYNV25VT4AFQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.085651679 (530h15m23.956734724s) [noise=true , address=45.91.116.9:4261     , piece_id=4ARIG7ZKLRH34DX6G5CQL5ZPSJWSHTWOEDMZEY6VWPWTPWI4UOMQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.116276664 (530h15m23.987359709s) [noise=true , address=77.90.31.19:4841     , piece_id=VVTSOM57CMQM4QGGMWOYXQ3LI7YXUQFIFNW6LJMH23Y5DSLR52IQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.143078408 (530h15m24.014161453s) [noise=true , address=77.90.31.14:4664     , piece_id=RIY2SZVO2D2YM3LLSHBQXM3PBGDFNEIJCSVWO264C2IY3DA6FSLA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.161306428 (530h15m24.032389473s) [noise=true , address=45.91.116.11:4119    , piece_id=PNXVSNSLCH6ZLU7BDGAT7JCHP6JGX2B5ODJDROXPMPTLMR3YGI7Q, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.835346707 (530h15m24.706429742s) [noise=true , address=45.91.116.14:4190    , piece_id=QOV7NJA6FODYGC6X2F63OQCSYV4ITY3VFJF6SH4DB2SEA4GI5XKQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841033896 (530h15m24.712116961s) [noise=true , address=77.90.31.11:4555     , piece_id=LASJXFQZNE5QVIBUYJUYH267C7O66NOBFNNZNELGBBSBUXPNYFRA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: <nil>
Dec 27 17:44:24.841058493 (530h15m24.712141528s) [seg_pos={0 0}, upload=465039]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:44:24.841288642 (530h15m24.712371717s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841298380 (530h15m24.712381425s) [noise=true , address=45.91.116.4:4939     , piece_id=PVHNAANWFEBHZ4IUGOTCYFDFGGAQQN2NWTGOJLX4YHVCVSQGCNKA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841476038 (530h15m24.712559093s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43774->145.79.227.3:4149: use of closed network connection
Dec 27 17:44:24.841485446 (530h15m24.712568491s) [noise=true , address=145.79.227.3:4149    , piece_id=BCZYWCMGTCKKFRW6DDZYJO6WI66THCSJ6ORN6JFPXGASIPAFOWFA, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841682682 (530h15m24.712765727s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 27 17:44:24.841692280 (530h15m24.712775315s) [noise=true , address=145.79.227.43:4157   , piece_id=6GXRUTZUNVM2PCPBQHNWZPPZONSMUZXCGM3LEO4JIFPZON62OBHQ, seg_pos={0 0}, upload=465039]: Long tail cancellation due to optimal threshold
Dec 27 17:44:24.841713280 (530h15m24.712796315s) [seg_pos={0 0}, upload=465039]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:44:24.841851554 (530h15m24.712934599s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:44:24.841859619 (530h15m24.712942664s) [upload=465039]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:44:25.141975962 (530h15m25.013059007s) [upload=465039]: Done uploading segments...
Dec 27 17:44:25.141997412 (530h15m25.013080447s) [upload=465039]: Upload finished. err: <nil>
Duration: 30.002290568s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................................................X.]
duration: 
3.535924861s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:44:33 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.535949306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................................................X]
duration: 
5.000762154s
trace: 
https://jaeger.dev.storj.io/trace/3bddfef0dfa96557
FAIL (2025-12-27 17:44:02 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000786509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....XX]
duration: 
30.003668692s
trace: 
https://jaeger.dev.storj.io/trace/2a9c5121044af59f
FAIL (2025-12-27 17:43:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 17:43:09.255000059 (530h14m9.126083104s) [upload=465024]: Starting upload
Dec 27 17:43:09.255077727 (530h14m9.126160772s) [upload=465024]: Done starting upload
Dec 27 17:43:09.255224135 (530h14m9.126307181s) [upload=465024]: Uploading segments...
Dec 27 17:43:09.255369392 (530h14m9.126452437s) [upload=465024]: Got next segment. Inline: false
Dec 27 17:43:09.255421101 (530h14m9.126504146s) [upload=465024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:43:09.255426701 (530h14m9.126509746s) [upload=465024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:43:09.508610938 (530h14m9.379693993s) [seg_pos={0 0}, upload=465024]: Begin upload segment...
Dec 27 17:43:09.508667125 (530h14m9.37975017s) [seg_pos={0 0}, upload=465024]: Done begin upload segment.
Dec 27 17:43:09.508696040 (530h14m9.379779075s) [noise=true , address=45.91.116.12:4225    , piece_id=AUVW6G4NWQ3MIGHKIGAN4G7TQZXWMGV72L24XUN226H4Y34U2KRQ, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.508823452 (530h14m9.379906507s) [noise=true , address=45.91.116.6:4974     , piece_id=KCKYTOZN644RQCIMZNO2LJTLHDPVPF7AHEEAS24VM7EURQEGC6EQ, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.509573781 (530h14m9.380656827s) [noise=true , address=145.79.227.15:4874   , piece_id=4FNNMQ3TDZ464HVZO2KMMFBRZDB3ZSBVH4ISAQBQX3NEXVQKB57A, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.509964305 (530h14m9.38104736s) [noise=true , address=145.79.227.42:4122   , piece_id=7J7BYA5YXV5FRGI7PYG6PCV5NA5DUJ4DQGG42H6ERRSV7CJ37YPQ, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.510485207 (530h14m9.381568262s) [noise=true , address=145.79.227.51:4409   , piece_id=QMV4GYVNI7ZFJSSAYXULI6XGFKUJ5DHUEGVUFX5MZ3W3KCYBYVRA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.510918502 (530h14m9.382001537s) [noise=true , address=77.90.31.15:4700     , piece_id=YACZ6UASF7WRF5T6C7ERCOZGIOPXKPA2RGS6EFYTK2E3Q5QN5UCA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.511297373 (530h14m9.382380418s) [noise=true , address=45.91.116.4:4937     , piece_id=MYRIPKDNCYLYG2PMMN5YNGYK73XZVWNRLFQLSSNFBMFTJIXDCOWA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.511572928 (530h14m9.382655963s) [noise=true , address=145.79.227.22:4509   , piece_id=WYN2PVX7XE24SEJBKJER67DRBXOBBG3LQSPGOHWXEZ6NERILDMNA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.511842561 (530h14m9.382925596s) [noise=true , address=145.79.227.36:4519   , piece_id=QN27ZH2XSPAYTR6FXWACCZXB3PEY5YHQULTL3HTXKEKOEGA3HQ7A, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.512121573 (530h14m9.383204608s) [noise=true , address=145.79.227.16:4910   , piece_id=275CEHB3DMKBVPJVML6GLVYQ4KMIOYGRUURR4AJLZPJPZREPVAMA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.512529380 (530h14m9.383612425s) [noise=true , address=145.79.227.9:4366    , piece_id=32MTE4SKWMPROBFOIN3U2XHQR2QQWFUP5HL5DJXOAN4FBJ7FCCJQ, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.513123491 (530h14m9.384206526s) [noise=true , address=77.90.31.12:4590     , piece_id=RN5RGBUSOWZISJIYAZDEI3MJLZDKVYXVOOTQ7DSMIAPOVTQEWJMQ, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.513611450 (530h14m9.384694495s) [noise=true , address=45.91.116.9:4263     , piece_id=WVN2PNALJT7KD5YEIKWZLL6YCPAVJZKPM4ZNVDVW6S4WVUPLHWKA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.514087787 (530h14m9.385170832s) [noise=true , address=145.79.227.32:4834   , piece_id=5KC6O3W2QE7VEURHGUDKWHKT3WHQUJ33ZD26M5UCJ2AT7XHPMYZA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.514602156 (530h14m9.385685191s) [noise=true , address=145.79.227.48:4301   , piece_id=BWLOVZI6NZMJHIMXZNOE7GKDR2AFSTTCXG5DDKI45VLWPIABAD5Q, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:09.515018138 (530h14m9.386101193s) [noise=true , address=145.79.227.11:4729   , piece_id=THSD5MDF76OJ5KX5VWDMKJNIBCTHRLEMAZNHXSZ3ZJUBHEF5PLFA, seg_pos={0 0}, upload=465024]: Uploading piece...
Dec 27 17:43:11.227631168 (530h14m11.098714203s) [upload=465024]: Next returned nil segment
Dec 27 17:43:11.227670954 (530h14m11.098753999s) [upload=465024]: Waiting for error group managing segments...
Dec 27 17:43:11.506332503 (530h14m11.377415548s) [noise=true , address=145.79.227.11:4729   , piece_id=THSD5MDF76OJ5KX5VWDMKJNIBCTHRLEMAZNHXSZ3ZJUBHEF5PLFA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.515018754 (530h14m11.386101799s) [noise=true , address=145.79.227.9:4366    , piece_id=32MTE4SKWMPROBFOIN3U2XHQR2QQWFUP5HL5DJXOAN4FBJ7FCCJQ, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.517140083 (530h14m11.388223128s) [noise=true , address=145.79.227.36:4519   , piece_id=QN27ZH2XSPAYTR6FXWACCZXB3PEY5YHQULTL3HTXKEKOEGA3HQ7A, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.517940127 (530h14m11.389023172s) [noise=true , address=145.79.227.51:4409   , piece_id=QMV4GYVNI7ZFJSSAYXULI6XGFKUJ5DHUEGVUFX5MZ3W3KCYBYVRA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.521901089 (530h14m11.392984134s) [noise=true , address=145.79.227.32:4834   , piece_id=5KC6O3W2QE7VEURHGUDKWHKT3WHQUJ33ZD26M5UCJ2AT7XHPMYZA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.526635985 (530h14m11.397719031s) [noise=true , address=145.79.227.15:4874   , piece_id=4FNNMQ3TDZ464HVZO2KMMFBRZDB3ZSBVH4ISAQBQX3NEXVQKB57A, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.529241818 (530h14m11.400324863s) [noise=true , address=145.79.227.42:4122   , piece_id=7J7BYA5YXV5FRGI7PYG6PCV5NA5DUJ4DQGG42H6ERRSV7CJ37YPQ, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.529896854 (530h14m11.400979899s) [noise=true , address=145.79.227.48:4301   , piece_id=BWLOVZI6NZMJHIMXZNOE7GKDR2AFSTTCXG5DDKI45VLWPIABAD5Q, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.544756710 (530h14m11.415839745s) [noise=true , address=145.79.227.22:4509   , piece_id=WYN2PVX7XE24SEJBKJER67DRBXOBBG3LQSPGOHWXEZ6NERILDMNA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.599786316 (530h14m11.470869361s) [noise=true , address=145.79.227.16:4910   , piece_id=275CEHB3DMKBVPJVML6GLVYQ4KMIOYGRUURR4AJLZPJPZREPVAMA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.811021173 (530h14m11.682104209s) [noise=true , address=45.91.116.6:4974     , piece_id=KCKYTOZN644RQCIMZNO2LJTLHDPVPF7AHEEAS24VM7EURQEGC6EQ, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.855131938 (530h14m11.726215003s) [noise=true , address=77.90.31.12:4590     , piece_id=RN5RGBUSOWZISJIYAZDEI3MJLZDKVYXVOOTQ7DSMIAPOVTQEWJMQ, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.877619228 (530h14m11.748702273s) [noise=true , address=45.91.116.4:4937     , piece_id=MYRIPKDNCYLYG2PMMN5YNGYK73XZVWNRLFQLSSNFBMFTJIXDCOWA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: <nil>
Dec 27 17:43:11.877646049 (530h14m11.748729094s) [seg_pos={0 0}, upload=465024]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:43:11.877962521 (530h14m11.749045566s) [noise=true , address=77.90.31.15:4700     , piece_id=YACZ6UASF7WRF5T6C7ERCOZGIOPXKPA2RGS6EFYTK2E3Q5QN5UCA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 27 17:43:11.877971729 (530h14m11.749054794s) [noise=true , address=77.90.31.15:4700     , piece_id=YACZ6UASF7WRF5T6C7ERCOZGIOPXKPA2RGS6EFYTK2E3Q5QN5UCA, seg_pos={0 0}, upload=465024]: Long tail cancellation due to optimal threshold
Dec 27 17:43:11.878108118 (530h14m11.749191163s) [noise=true , address=45.91.116.9:4263     , piece_id=WVN2PNALJT7KD5YEIKWZLL6YCPAVJZKPM4ZNVDVW6S4WVUPLHWKA, seg_pos={0 0}, upload=465024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 17:43:11.878135180 (530h14m11.749218215s) [noise=true , address=45.91.116.9:4263     , piece_id=WVN2PNALJT7KD5YEIKWZLL6YCPAVJZKPM4ZNVDVW6S4WVUPLHWKA, seg_pos={0 0}, upload=465024]: Long tail cancellation due to optimal threshold
Dec 27 17:43:11.878317066 (530h14m11.749400141s) [noise=true , address=45.91.116.12:4225    , piece_id=AUVW6G4NWQ3MIGHKIGAN4G7TQZXWMGV72L24XUN226H4Y34U2KRQ, seg_pos={0 0}, upload=465024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 27 17:43:11.878328758 (530h14m11.749411803s) [noise=true , address=45.91.116.12:4225    , piece_id=AUVW6G4NWQ3MIGHKIGAN4G7TQZXWMGV72L24XUN226H4Y34U2KRQ, seg_pos={0 0}, upload=465024]: Long tail cancellation due to optimal threshold
Dec 27 17:43:11.878354878 (530h14m11.749437913s) [seg_pos={0 0}, upload=465024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:43:11.878574928 (530h14m11.749657973s) [upload=465024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:43:11.878586620 (530h14m11.749669665s) [upload=465024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:43:12.171420706 (530h14m12.042503751s) [upload=465024]: Done uploading segments...
Dec 27 17:43:12.171439431 (530h14m12.042522466s) [upload=465024]: Upload finished. err: <nil>
Duration: 30.003696985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [.X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....X]
duration: 
30.002597393s
trace: 
https://jaeger.dev.storj.io/trace/690b9c2e5c660afb
FAIL (2025-12-27 17:42:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EENBVLJ44I6X4EWASENKP2CUVJJETGDNUDQVAW3PE4B6BC26R5DQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: LRDXUXIWJHFPEIW532MA6CJJHSCAC7ORVJLDU7ZMRCEQ4VWG3YUQ) context deadline exceeded
======LOGS======
Dec 27 17:42:03.272402427 (530h13m3.143485472s) [upload=465009]: Starting upload
Dec 27 17:42:03.272468944 (530h13m3.143551979s) [upload=465009]: Done starting upload
Dec 27 17:42:03.272695835 (530h13m3.14377888s) [upload=465009]: Uploading segments...
Dec 27 17:42:03.272861661 (530h13m3.143944737s) [upload=465009]: Got next segment. Inline: false
Dec 27 17:42:03.272915094 (530h13m3.143998149s) [upload=465009]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:42:03.272922327 (530h13m3.144005372s) [upload=465009]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:42:03.518363941 (530h13m3.389446996s) [seg_pos={0 0}, upload=465009]: Begin upload segment...
Dec 27 17:42:03.518540036 (530h13m3.389623081s) [seg_pos={0 0}, upload=465009]: Done begin upload segment.
Dec 27 17:42:03.518584661 (530h13m3.389667706s) [noise=true , address=145.79.227.37:4565   , piece_id=3CUOAGSUD5GZSE4WH6CYQBDQ7KCFJFXNK62WEXA75JGDQUA4YFRQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.518671787 (530h13m3.389754862s) [noise=true , address=145.79.227.52:4446   , piece_id=DLHGHHEV7CXGH2DGKQL5BW3MJBXVLRY64TSX76IKSZ45TB4NSKVA, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.519359436 (530h13m3.390442481s) [noise=true , address=45.91.116.4:4940     , piece_id=KL4H2ML27TDXKXJSU3W3ZFBZK2QHYALE3ZJQWYRG7QHQUL5M4Y2A, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.519797971 (530h13m3.390881016s) [noise=true , address=145.79.227.10:4403   , piece_id=Q3UVQOWV6FBTYEZNZPXMOWS6ECROIHNINZVELTWHQMOATI6ME5KQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.520171102 (530h13m3.391254147s) [noise=true , address=145.79.227.11:4732   , piece_id=NBH5V7ZT5ZJCSWMPRTOL44BPIEUCBDTXII36MJRUPOMPSTAE6M4Q, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.520198012 (530h13m3.391281057s) [noise=true , address=77.90.31.15:4697     , piece_id=2GPOWXBSYSBKSRKA2VF3WY4PHDWUVW7FBEWEDDE4CQU2OSXPDN2A, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.520605549 (530h13m3.391688594s) [noise=true , address=145.79.227.24:4581   , piece_id=EWGPNJCZ3HENGJMWGJGRYXV4EZVGOCA22ILTPVTJDU2DF6GUTU4A, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.520920348 (530h13m3.392003403s) [noise=true , address=145.79.227.14:4839   , piece_id=TGGZBXCDZBZ5RHK6HEU4DTC5DDQTO54Q6ZBKXRWKX7QESV6YUFNA, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.521021120 (530h13m3.392104165s) [noise=true , address=45.91.116.7:4012     , piece_id=TO67EHKCQKZH53WPOORFDDSBONCRBJL6WTRLPMZWGTQE2G2JEFZQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.521444095 (530h13m3.39252715s) [noise=true , address=77.90.31.10:4519     , piece_id=EENBVLJ44I6X4EWASENKP2CUVJJETGDNUDQVAW3PE4B6BC26R5DQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.521625280 (530h13m3.392708325s) [noise=true , address=45.91.116.5:4300     , piece_id=LRDXUXIWJHFPEIW532MA6CJJHSCAC7ORVJLDU7ZMRCEQ4VWG3YUQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.521921565 (530h13m3.39300461s) [noise=true , address=145.79.227.41:4087   , piece_id=4S32VZXOQ5BGCOLRDH6SRIJCX6SHIUFNFO6Z2UL2THFMPDNR7EYA, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.522339561 (530h13m3.393422596s) [noise=true , address=145.79.227.26:4653   , piece_id=BQO6B7HPQVA6S2SXGGJH5PIVZ5ML5A56V5QCM6RCI3MFKUOW3QXQ, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.522400447 (530h13m3.393483492s) [noise=true , address=145.79.227.22:4509   , piece_id=PBI6BK3QNAJKC3HGSOY4ZIOG42GKWUXTD5NTWB7ECN4X4ZO4L5RA, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.523378279 (530h13m3.394461364s) [noise=true , address=145.79.227.47:4265   , piece_id=QULVYMQ5HBN2RJ3KQ4K4MRJYGV6XSO72VJNSXLV5AFCCZ4UKD47Q, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:03.524284314 (530h13m3.395367359s) [noise=true , address=145.79.227.15:4876   , piece_id=6S44NOT73BA7I2WL6IPST5M3I6SODQJMSBMEJIQWL7IVVJEYPFVA, seg_pos={0 0}, upload=465009]: Uploading piece...
Dec 27 17:42:05.145849043 (530h13m5.016932088s) [upload=465009]: Next returned nil segment
Dec 27 17:42:05.145880633 (530h13m5.016963698s) [upload=465009]: Waiting for error group managing segments...
Dec 27 17:42:05.527273169 (530h13m5.398356214s) [noise=true , address=145.79.227.37:4565   , piece_id=3CUOAGSUD5GZSE4WH6CYQBDQ7KCFJFXNK62WEXA75JGDQUA4YFRQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.527808199 (530h13m5.398891234s) [noise=true , address=145.79.227.15:4876   , piece_id=6S44NOT73BA7I2WL6IPST5M3I6SODQJMSBMEJIQWL7IVVJEYPFVA, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.530995237 (530h13m5.402078282s) [noise=true , address=145.79.227.10:4403   , piece_id=Q3UVQOWV6FBTYEZNZPXMOWS6ECROIHNINZVELTWHQMOATI6ME5KQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.531186832 (530h13m5.402269877s) [noise=true , address=145.79.227.52:4446   , piece_id=DLHGHHEV7CXGH2DGKQL5BW3MJBXVLRY64TSX76IKSZ45TB4NSKVA, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.532038072 (530h13m5.403121107s) [noise=true , address=145.79.227.41:4087   , piece_id=4S32VZXOQ5BGCOLRDH6SRIJCX6SHIUFNFO6Z2UL2THFMPDNR7EYA, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.542571881 (530h13m5.413654916s) [noise=true , address=145.79.227.14:4839   , piece_id=TGGZBXCDZBZ5RHK6HEU4DTC5DDQTO54Q6ZBKXRWKX7QESV6YUFNA, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.544187046 (530h13m5.415270081s) [noise=true , address=145.79.227.26:4653   , piece_id=BQO6B7HPQVA6S2SXGGJH5PIVZ5ML5A56V5QCM6RCI3MFKUOW3QXQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.550752088 (530h13m5.421835133s) [noise=true , address=145.79.227.22:4509   , piece_id=PBI6BK3QNAJKC3HGSOY4ZIOG42GKWUXTD5NTWB7ECN4X4ZO4L5RA, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.560548962 (530h13m5.431632007s) [noise=true , address=145.79.227.24:4581   , piece_id=EWGPNJCZ3HENGJMWGJGRYXV4EZVGOCA22ILTPVTJDU2DF6GUTU4A, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.561204150 (530h13m5.432287195s) [noise=true , address=145.79.227.11:4732   , piece_id=NBH5V7ZT5ZJCSWMPRTOL44BPIEUCBDTXII36MJRUPOMPSTAE6M4Q, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.563174242 (530h13m5.434257287s) [noise=true , address=145.79.227.47:4265   , piece_id=QULVYMQ5HBN2RJ3KQ4K4MRJYGV6XSO72VJNSXLV5AFCCZ4UKD47Q, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.693280346 (530h13m5.564363391s) [noise=true , address=77.90.31.10:4519     , piece_id=EENBVLJ44I6X4EWASENKP2CUVJJETGDNUDQVAW3PE4B6BC26R5DQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.779370736 (530h13m5.650453782s) [noise=true , address=45.91.116.5:4300     , piece_id=LRDXUXIWJHFPEIW532MA6CJJHSCAC7ORVJLDU7ZMRCEQ4VWG3YUQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: <nil>
Dec 27 17:42:05.779406625 (530h13m5.65048967s) [seg_pos={0 0}, upload=465009]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:42:05.779671670 (530h13m5.650754715s) [noise=true , address=77.90.31.15:4697     , piece_id=2GPOWXBSYSBKSRKA2VF3WY4PHDWUVW7FBEWEDDE4CQU2OSXPDN2A, seg_pos={0 0}, upload=465009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 27 17:42:05.779682019 (530h13m5.650765084s) [noise=true , address=77.90.31.15:4697     , piece_id=2GPOWXBSYSBKSRKA2VF3WY4PHDWUVW7FBEWEDDE4CQU2OSXPDN2A, seg_pos={0 0}, upload=465009]: Long tail cancellation due to optimal threshold
Dec 27 17:42:05.779711225 (530h13m5.65079428s) [noise=true , address=45.91.116.4:4940     , piece_id=KL4H2ML27TDXKXJSU3W3ZFBZK2QHYALE3ZJQWYRG7QHQUL5M4Y2A, seg_pos={0 0}, upload=465009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 27 17:42:05.779721365 (530h13m5.65080441s) [noise=true , address=45.91.116.4:4940     , piece_id=KL4H2ML27TDXKXJSU3W3ZFBZK2QHYALE3ZJQWYRG7QHQUL5M4Y2A, seg_pos={0 0}, upload=465009]: Long tail cancellation due to optimal threshold
Dec 27 17:42:05.779998913 (530h13m5.651081968s) [noise=true , address=45.91.116.7:4012     , piece_id=TO67EHKCQKZH53WPOORFDDSBONCRBJL6WTRLPMZWGTQE2G2JEFZQ, seg_pos={0 0}, upload=465009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: context canceled; context canceled
Dec 27 17:42:05.780009433 (530h13m5.651092478s) [noise=true , address=45.91.116.7:4012     , piece_id=TO67EHKCQKZH53WPOORFDDSBONCRBJL6WTRLPMZWGTQE2G2JEFZQ, seg_pos={0 0}, upload=465009]: Long tail cancellation due to optimal threshold
Dec 27 17:42:05.780028890 (530h13m5.651111935s) [seg_pos={0 0}, upload=465009]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:42:05.780114143 (530h13m5.651197178s) [upload=465009]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:42:05.780134030 (530h13m5.651217075s) [upload=465009]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:42:06.072426566 (530h13m5.943509601s) [upload=465009]: Done uploading segments...
Dec 27 17:42:06.072446113 (530h13m5.943529359s) [upload=465009]: Upload finished. err: <nil>
Duration: 30.002625237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [..X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X....]
duration: 
5.887051359s
trace: 
https://jaeger.dev.storj.io/trace/6a0d03aa4e18f3ee
FAIL (2025-12-27 17:40:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ) context deadline exceeded
======LOGS======
56.267423542s) [upload=464941]: Uploading segments...
Dec 27 17:36:56.396437241 (530h7m56.267520276s) [upload=464941]: Got next segment. Inline: false
Dec 27 17:36:56.396472438 (530h7m56.267555483s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:36:56.396477708 (530h7m56.267560743s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:36:56.644074756 (530h7m56.515157801s) [seg_pos={0 0}, upload=464941]: Begin upload segment...
Dec 27 17:36:56.644130853 (530h7m56.515213888s) [seg_pos={0 0}, upload=464941]: Done begin upload segment.
Dec 27 17:36:56.644174125 (530h7m56.51525717s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644228688 (530h7m56.515311724s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644695638 (530h7m56.515778683s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645135305 (530h7m56.51621835s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645567619 (530h7m56.516650664s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646022485 (530h7m56.51710553s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646479835 (530h7m56.51756288s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646906217 (530h7m56.517989262s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648233124 (530h7m56.519316159s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648396836 (530h7m56.519479871s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.649515846 (530h7m56.520598881s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.651353756 (530h7m56.522436801s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.653018496 (530h7m56.524101541s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.654628031 (530h7m56.525711066s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.656792773 (530h7m56.527875808s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.657824167 (530h7m56.528907202s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:58.352425772 (530h7m58.223508817s) [upload=464941]: Next returned nil segment
Dec 27 17:36:58.352444477 (530h7m58.223527522s) [upload=464941]: Waiting for error group managing segments...
Dec 27 17:36:58.640222305 (530h7m58.511305351s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.663137139 (530h7m58.534220184s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.666274764 (530h7m58.5373578s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670449704 (530h7m58.541532739s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670936661 (530h7m58.542019696s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.682997788 (530h7m58.554080823s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.862846350 (530h7m58.733929395s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.932018246 (530h7m58.803101291s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.933552227 (530h7m58.804635272s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.951192065 (530h7m58.822275111s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.204116548 (530h7m59.075199593s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.205388240 (530h7m59.076471275s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617215569 (530h7m59.488298605s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617235658 (530h7m59.488318703s) [seg_pos={0 0}, upload=464941]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:36:59.617409149 (530h7m59.488492184s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection
Dec 27 17:36:59.617414899 (530h7m59.488497934s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617624228 (530h7m59.488707273s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection
Dec 27 17:36:59.617631251 (530h7m59.488714286s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617722485 (530h7m59.48880553s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF
Dec 27 17:36:59.617727655 (530h7m59.48881069s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617741030 (530h7m59.488824075s) [seg_pos={0 0}, upload=464941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:36:59.617832454 (530h7m59.488915499s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:36:59.617838255 (530h7m59.48892129s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:36:59.909123864 (530h7m59.780206909s) [upload=464941]: Done uploading segments...
Dec 27 17:36:59.909141877 (530h7m59.780224913s) [upload=464941]: Upload finished. err: <nil>
Duration: 5.887112727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [...X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X...]
duration: 
6.127044749s
trace: 
https://jaeger.dev.storj.io/trace/6a0d03aa4e18f3ee
FAIL (2025-12-27 17:40:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ) context deadline exceeded
======LOGS======
56.267423542s) [upload=464941]: Uploading segments...
Dec 27 17:36:56.396437241 (530h7m56.267520276s) [upload=464941]: Got next segment. Inline: false
Dec 27 17:36:56.396472438 (530h7m56.267555483s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:36:56.396477708 (530h7m56.267560743s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:36:56.644074756 (530h7m56.515157801s) [seg_pos={0 0}, upload=464941]: Begin upload segment...
Dec 27 17:36:56.644130853 (530h7m56.515213888s) [seg_pos={0 0}, upload=464941]: Done begin upload segment.
Dec 27 17:36:56.644174125 (530h7m56.51525717s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644228688 (530h7m56.515311724s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644695638 (530h7m56.515778683s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645135305 (530h7m56.51621835s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645567619 (530h7m56.516650664s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646022485 (530h7m56.51710553s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646479835 (530h7m56.51756288s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646906217 (530h7m56.517989262s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648233124 (530h7m56.519316159s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648396836 (530h7m56.519479871s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.649515846 (530h7m56.520598881s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.651353756 (530h7m56.522436801s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.653018496 (530h7m56.524101541s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.654628031 (530h7m56.525711066s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.656792773 (530h7m56.527875808s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.657824167 (530h7m56.528907202s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:58.352425772 (530h7m58.223508817s) [upload=464941]: Next returned nil segment
Dec 27 17:36:58.352444477 (530h7m58.223527522s) [upload=464941]: Waiting for error group managing segments...
Dec 27 17:36:58.640222305 (530h7m58.511305351s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.663137139 (530h7m58.534220184s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.666274764 (530h7m58.5373578s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670449704 (530h7m58.541532739s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670936661 (530h7m58.542019696s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.682997788 (530h7m58.554080823s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.862846350 (530h7m58.733929395s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.932018246 (530h7m58.803101291s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.933552227 (530h7m58.804635272s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.951192065 (530h7m58.822275111s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.204116548 (530h7m59.075199593s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.205388240 (530h7m59.076471275s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617215569 (530h7m59.488298605s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617235658 (530h7m59.488318703s) [seg_pos={0 0}, upload=464941]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:36:59.617409149 (530h7m59.488492184s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection
Dec 27 17:36:59.617414899 (530h7m59.488497934s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617624228 (530h7m59.488707273s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection
Dec 27 17:36:59.617631251 (530h7m59.488714286s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617722485 (530h7m59.48880553s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF
Dec 27 17:36:59.617727655 (530h7m59.48881069s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617741030 (530h7m59.488824075s) [seg_pos={0 0}, upload=464941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:36:59.617832454 (530h7m59.488915499s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:36:59.617838255 (530h7m59.48892129s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:36:59.909123864 (530h7m59.780206909s) [upload=464941]: Done uploading segments...
Dec 27 17:36:59.909141877 (530h7m59.780224913s) [upload=464941]: Upload finished. err: <nil>
Duration: 6.127083642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [....X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X..]
duration: 
5.980296435s
trace: 
https://jaeger.dev.storj.io/trace/6a0d03aa4e18f3ee
FAIL (2025-12-27 17:39:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ) context deadline exceeded
======LOGS======
56.267423542s) [upload=464941]: Uploading segments...
Dec 27 17:36:56.396437241 (530h7m56.267520276s) [upload=464941]: Got next segment. Inline: false
Dec 27 17:36:56.396472438 (530h7m56.267555483s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:36:56.396477708 (530h7m56.267560743s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:36:56.644074756 (530h7m56.515157801s) [seg_pos={0 0}, upload=464941]: Begin upload segment...
Dec 27 17:36:56.644130853 (530h7m56.515213888s) [seg_pos={0 0}, upload=464941]: Done begin upload segment.
Dec 27 17:36:56.644174125 (530h7m56.51525717s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644228688 (530h7m56.515311724s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644695638 (530h7m56.515778683s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645135305 (530h7m56.51621835s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645567619 (530h7m56.516650664s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646022485 (530h7m56.51710553s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646479835 (530h7m56.51756288s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646906217 (530h7m56.517989262s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648233124 (530h7m56.519316159s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648396836 (530h7m56.519479871s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.649515846 (530h7m56.520598881s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.651353756 (530h7m56.522436801s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.653018496 (530h7m56.524101541s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.654628031 (530h7m56.525711066s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.656792773 (530h7m56.527875808s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.657824167 (530h7m56.528907202s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:58.352425772 (530h7m58.223508817s) [upload=464941]: Next returned nil segment
Dec 27 17:36:58.352444477 (530h7m58.223527522s) [upload=464941]: Waiting for error group managing segments...
Dec 27 17:36:58.640222305 (530h7m58.511305351s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.663137139 (530h7m58.534220184s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.666274764 (530h7m58.5373578s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670449704 (530h7m58.541532739s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670936661 (530h7m58.542019696s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.682997788 (530h7m58.554080823s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.862846350 (530h7m58.733929395s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.932018246 (530h7m58.803101291s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.933552227 (530h7m58.804635272s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.951192065 (530h7m58.822275111s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.204116548 (530h7m59.075199593s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.205388240 (530h7m59.076471275s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617215569 (530h7m59.488298605s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617235658 (530h7m59.488318703s) [seg_pos={0 0}, upload=464941]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:36:59.617409149 (530h7m59.488492184s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection
Dec 27 17:36:59.617414899 (530h7m59.488497934s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617624228 (530h7m59.488707273s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection
Dec 27 17:36:59.617631251 (530h7m59.488714286s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617722485 (530h7m59.48880553s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF
Dec 27 17:36:59.617727655 (530h7m59.48881069s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617741030 (530h7m59.488824075s) [seg_pos={0 0}, upload=464941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:36:59.617832454 (530h7m59.488915499s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:36:59.617838255 (530h7m59.48892129s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:36:59.909123864 (530h7m59.780206909s) [upload=464941]: Done uploading segments...
Dec 27 17:36:59.909141877 (530h7m59.780224913s) [upload=464941]: Upload finished. err: <nil>
Duration: 5.980337142s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [X...........................................................]
duration: 
3.39373169s
trace: 
https://jaeger.dev.storj.io/trace/63f3b817fe611960
FAIL (2025-12-27 17:38:51 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.393763049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.....X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X.]
duration: 
5.891121441s
trace: 
https://jaeger.dev.storj.io/trace/6a0d03aa4e18f3ee
FAIL (2025-12-27 17:38:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ) context deadline exceeded
======LOGS======
56.267423542s) [upload=464941]: Uploading segments...
Dec 27 17:36:56.396437241 (530h7m56.267520276s) [upload=464941]: Got next segment. Inline: false
Dec 27 17:36:56.396472438 (530h7m56.267555483s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:36:56.396477708 (530h7m56.267560743s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:36:56.644074756 (530h7m56.515157801s) [seg_pos={0 0}, upload=464941]: Begin upload segment...
Dec 27 17:36:56.644130853 (530h7m56.515213888s) [seg_pos={0 0}, upload=464941]: Done begin upload segment.
Dec 27 17:36:56.644174125 (530h7m56.51525717s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644228688 (530h7m56.515311724s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644695638 (530h7m56.515778683s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645135305 (530h7m56.51621835s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645567619 (530h7m56.516650664s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646022485 (530h7m56.51710553s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646479835 (530h7m56.51756288s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646906217 (530h7m56.517989262s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648233124 (530h7m56.519316159s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648396836 (530h7m56.519479871s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.649515846 (530h7m56.520598881s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.651353756 (530h7m56.522436801s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.653018496 (530h7m56.524101541s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.654628031 (530h7m56.525711066s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.656792773 (530h7m56.527875808s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.657824167 (530h7m56.528907202s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:58.352425772 (530h7m58.223508817s) [upload=464941]: Next returned nil segment
Dec 27 17:36:58.352444477 (530h7m58.223527522s) [upload=464941]: Waiting for error group managing segments...
Dec 27 17:36:58.640222305 (530h7m58.511305351s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.663137139 (530h7m58.534220184s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.666274764 (530h7m58.5373578s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670449704 (530h7m58.541532739s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670936661 (530h7m58.542019696s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.682997788 (530h7m58.554080823s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.862846350 (530h7m58.733929395s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.932018246 (530h7m58.803101291s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.933552227 (530h7m58.804635272s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.951192065 (530h7m58.822275111s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.204116548 (530h7m59.075199593s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.205388240 (530h7m59.076471275s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617215569 (530h7m59.488298605s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617235658 (530h7m59.488318703s) [seg_pos={0 0}, upload=464941]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:36:59.617409149 (530h7m59.488492184s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection
Dec 27 17:36:59.617414899 (530h7m59.488497934s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617624228 (530h7m59.488707273s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection
Dec 27 17:36:59.617631251 (530h7m59.488714286s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617722485 (530h7m59.48880553s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF
Dec 27 17:36:59.617727655 (530h7m59.48881069s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617741030 (530h7m59.488824075s) [seg_pos={0 0}, upload=464941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:36:59.617832454 (530h7m59.488915499s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:36:59.617838255 (530h7m59.48892129s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:36:59.909123864 (530h7m59.780206909s) [upload=464941]: Done uploading segments...
Dec 27 17:36:59.909141877 (530h7m59.780224913s) [upload=464941]: Upload finished. err: <nil>
Duration: 5.891219747s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.X..........................................................]
duration: 
3.603759398s
trace: 
https://jaeger.dev.storj.io/trace/63f3b817fe611960
FAIL (2025-12-27 17:37:45 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.603774397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [......X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.X]
duration: 
30.005661276s
trace: 
https://jaeger.dev.storj.io/trace/6a0d03aa4e18f3ee
FAIL (2025-12-27 17:37:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ) context deadline exceeded
======LOGS======
56.267423542s) [upload=464941]: Uploading segments...
Dec 27 17:36:56.396437241 (530h7m56.267520276s) [upload=464941]: Got next segment. Inline: false
Dec 27 17:36:56.396472438 (530h7m56.267555483s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:36:56.396477708 (530h7m56.267560743s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:36:56.644074756 (530h7m56.515157801s) [seg_pos={0 0}, upload=464941]: Begin upload segment...
Dec 27 17:36:56.644130853 (530h7m56.515213888s) [seg_pos={0 0}, upload=464941]: Done begin upload segment.
Dec 27 17:36:56.644174125 (530h7m56.51525717s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644228688 (530h7m56.515311724s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.644695638 (530h7m56.515778683s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645135305 (530h7m56.51621835s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.645567619 (530h7m56.516650664s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646022485 (530h7m56.51710553s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646479835 (530h7m56.51756288s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.646906217 (530h7m56.517989262s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648233124 (530h7m56.519316159s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.648396836 (530h7m56.519479871s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.649515846 (530h7m56.520598881s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.651353756 (530h7m56.522436801s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.653018496 (530h7m56.524101541s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.654628031 (530h7m56.525711066s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.656792773 (530h7m56.527875808s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:56.657824167 (530h7m56.528907202s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Uploading piece...
Dec 27 17:36:58.352425772 (530h7m58.223508817s) [upload=464941]: Next returned nil segment
Dec 27 17:36:58.352444477 (530h7m58.223527522s) [upload=464941]: Waiting for error group managing segments...
Dec 27 17:36:58.640222305 (530h7m58.511305351s) [noise=true , address=145.79.227.38:4977   , piece_id=NMXOILK3BIPWO7QYKUG2RFGVMHY36NDORCKJW36PE6NQTL66T77Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.663137139 (530h7m58.534220184s) [noise=true , address=145.79.227.2:4114    , piece_id=R3LYPX6BTHIKOGCJF2EFGT7AH4JDH63GU2MFOWRLGW7ISCCDBDCQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.666274764 (530h7m58.5373578s) [noise=true , address=145.79.227.9:4366    , piece_id=RQLEH67KBNX4RJUSCDDSZLTT3NPRVWOPTDY2LXBRRU43OP42LJAQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670449704 (530h7m58.541532739s) [noise=true , address=145.79.227.41:4086   , piece_id=KH4IV5XBARZWPQYLZO64SSAW4PVITRDAWCQH5MY7M3DUMQ3MUHYA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.670936661 (530h7m58.542019696s) [noise=true , address=145.79.227.11:4729   , piece_id=F4AONFXKBRH7IXJNK4DYQ345Q2WDSMUDNSJQFWGZITAY6BRTXNVQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.682997788 (530h7m58.554080823s) [noise=true , address=145.79.227.51:4409   , piece_id=GSXBXKNY5J5X7KPTKP7TBEMBMNIXZ6BQDPJ6RKP3T7JZVORNEVZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.862846350 (530h7m58.733929395s) [noise=true , address=45.91.116.11:4119    , piece_id=OWDYC7JXXTQ52D3DF2KSMF5HRE3OI7ZIXIK4RTTISEC3UAMAH6ZQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.932018246 (530h7m58.803101291s) [noise=true , address=45.91.116.14:4190    , piece_id=OO4YPOFEP74YMN26V2N7QPGU3IOY7JYHZ4I4OEFJYN5ZR66BSNPA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.933552227 (530h7m58.804635272s) [noise=true , address=45.91.116.7:4011     , piece_id=DL2H52HF6M4XRODK33TT6F7W6QKIVG6RLBKCMHSCYYOT5H6LVPXQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:58.951192065 (530h7m58.822275111s) [noise=true , address=77.90.31.19:4844     , piece_id=AQUUSICNNP7ZCC32G7MP6PBBCXNZIPGHAF37IKIKRSSQ4IBFVAKQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.204116548 (530h7m59.075199593s) [noise=true , address=45.91.116.4:4939     , piece_id=B73DLC4MVOAC47GAJYD4JBS5PTEE6ZJ7L53725UKWSWQKTPA5K3Q, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.205388240 (530h7m59.076471275s) [noise=true , address=45.91.116.9:4263     , piece_id=K6LEXQ354DKVYAE4QVER3QJVEQXUJM5IIOSVEGZLENFDSPV54IYQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617215569 (530h7m59.488298605s) [noise=true , address=77.90.31.11:4553     , piece_id=EQFHHNTSLCUUZ2IU2EZF24G66R2PF3VGCEGAGXJNDBH23C5U4CEQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: <nil>
Dec 27 17:36:59.617235658 (530h7m59.488318703s) [seg_pos={0 0}, upload=464941]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:36:59.617409149 (530h7m59.488492184s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->145.79.227.19:4078: use of closed network connection
Dec 27 17:36:59.617414899 (530h7m59.488497934s) [noise=true , address=145.79.227.19:4078   , piece_id=EBGPNT6UY3NDDVTPMKLCHNK6UOG6KYR2P7XTG56WPYBWE55536WQ, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617624228 (530h7m59.488707273s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->145.79.227.26:4655: use of closed network connection
Dec 27 17:36:59.617631251 (530h7m59.488714286s) [noise=true , address=145.79.227.26:4655   , piece_id=PRW7XSKD5EODQWN33YANDL4LCR4UFODTCLU3UBGEBHKUI4TKEZZA, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617722485 (530h7m59.48880553s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: expected piece hash; context canceled; EOF
Dec 27 17:36:59.617727655 (530h7m59.48881069s) [noise=true , address=145.79.227.17:4971   , piece_id=ALWE7RELP2I5TWKUX5PWUL2ULZIUCXAQ2WQV44R6KPFB5DJZYA7A, seg_pos={0 0}, upload=464941]: Long tail cancellation due to optimal threshold
Dec 27 17:36:59.617741030 (530h7m59.488824075s) [seg_pos={0 0}, upload=464941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:36:59.617832454 (530h7m59.488915499s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:36:59.617838255 (530h7m59.48892129s) [upload=464941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:36:59.909123864 (530h7m59.780206909s) [upload=464941]: Done uploading segments...
Dec 27 17:36:59.909141877 (530h7m59.780224913s) [upload=464941]: Upload finished. err: <nil>
Duration: 30.00568493s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..X.........................................................]
duration: 
3.60904769s
trace: 
https://jaeger.dev.storj.io/trace/63f3b817fe611960
FAIL (2025-12-27 17:37:12 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.609075954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...X........................................................]
duration: 
3.631376148s
trace: 
https://jaeger.dev.storj.io/trace/63f3b817fe611960
FAIL (2025-12-27 17:36:43 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.631413179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [X......X.....X.X...X...X...XXXX.X..XX.XX.XX.....X.X.X..XXXX.]
duration: 
7.47974569s
trace: 
https://jaeger.dev.storj.io/trace/8ee4103cb629be8
FAIL (2025-12-27 17:35:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 17:34:49.505055414 (530h5m49.376138449s) [upload=464907]: Starting upload
Dec 27 17:34:49.505133563 (530h5m49.376216608s) [upload=464907]: Done starting upload
Dec 27 17:34:49.505189560 (530h5m49.376272605s) [upload=464907]: Uploading segments...
Dec 27 17:34:49.505319186 (530h5m49.376402221s) [upload=464907]: Got next segment. Inline: false
Dec 27 17:34:49.505371226 (530h5m49.376454261s) [upload=464907]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 17:34:49.505377998 (530h5m49.376461043s) [upload=464907]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 17:34:49.766657029 (530h5m49.637740074s) [seg_pos={0 0}, upload=464907]: Begin upload segment...
Dec 27 17:34:49.766726722 (530h5m49.637809757s) [seg_pos={0 0}, upload=464907]: Done begin upload segment.
Dec 27 17:34:49.766756128 (530h5m49.637839163s) [noise=true , address=145.79.227.9:4366    , piece_id=6JAH4DWLDO32SKPGCUFO6PP6AFCQFELRQWZ6OIERJWNQ5NXDAW5A, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.766781116 (530h5m49.637864161s) [noise=true , address=145.79.227.14:4839   , piece_id=2NVCYCPU6BQUJT5SAATIITNETMLEQCNSSY5PLDE56RRTUKZSG6UA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.766844577 (530h5m49.637927622s) [noise=true , address=45.91.116.12:4227    , piece_id=EXCF4SMYHUQGX37EGNJLZF3I7FXMAOEM5SOIZELNHWLH2M2FAM6A, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.767200855 (530h5m49.6382839s) [noise=true , address=145.79.227.50:4375   , piece_id=DX5CF4WG6YZP6CB4K7V6ZK5KYVM4H7KWYXZPGFC4ZEL6OQUDDWQA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.767889066 (530h5m49.638972101s) [noise=true , address=145.79.227.4:4186    , piece_id=B7L44AGZT2ZCM4WILSZQBCWDR4WASEWMRYNKKIUII4S3NR376PPA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.768043760 (530h5m49.639126815s) [noise=true , address=45.91.116.7:4011     , piece_id=COPOL5ZS3Q2HAXP4SLWKA5WKAV5FEUZE645S2XTMPIVQ3Z5EUDHQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.768216078 (530h5m49.639299113s) [noise=true , address=145.79.227.48:4301   , piece_id=QGSQ7KUES6GWJ2UOW7UJ43WSLMPDDHQ555X46WSXEPKG3ZANCOEQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.768756327 (530h5m49.639839362s) [noise=true , address=145.79.227.32:4834   , piece_id=QSOOIRQHGTPQLDMDODL2Z3GNBB6M3TNZ5FD5R72MY33YZ6KTWTNA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.769231071 (530h5m49.640314116s) [noise=true , address=145.79.227.52:4447   , piece_id=Q3453I4PXSXFZSOQ7DBJNDFQDKBKQLTM7EF3FQW63P25FUMRCPDQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.769854167 (530h5m49.640937232s) [noise=true , address=45.91.116.6:4976     , piece_id=YBJ7QU3C6VE7OVKMHFA2DJ5IMSDDNL26NBBBGSNW4GKZJ4YLO5HQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.770200927 (530h5m49.641283963s) [noise=true , address=145.79.227.3:4151    , piece_id=7SQKKQTEETYFBG6KF6MF5RL6JP3J454IDF66SBPG42A2QU6AGEOA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.770816700 (530h5m49.641899735s) [noise=true , address=77.90.31.11:4556     , piece_id=OOANCVFEVLLG3LO2HU4WSPV6VXKEDSCEF3HOZF5PLU5TORYIXDQA, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.771637382 (530h5m49.642720428s) [noise=true , address=45.91.116.5:4298     , piece_id=QXYGO524EWSMLWVVHYSFRAB445BBMZWST5FKCFFYPDFEZFGESZYQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.772205024 (530h5m49.643288059s) [noise=true , address=145.79.227.17:4972   , piece_id=34JJIUEI3MZ6RYPOR7DR4WDRA4HU7T6KCLV3VLMRPDYP6VXODWTQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.772572784 (530h5m49.643655819s) [noise=true , address=145.79.227.39:4015   , piece_id=UVWN5AX4JDHSZQPJU6JR3XEX7L2XQ2H4DD5A2DNTD6ENQRXTTDOQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:49.773327922 (530h5m49.644410957s) [noise=true , address=145.79.227.10:4403   , piece_id=ZUUGIZ46V4O3F5TGYIGAWTFECYKOUHFBWE27QD2HL2EEXOZUTDWQ, seg_pos={0 0}, upload=464907]: Uploading piece...
Dec 27 17:34:51.472913578 (530h5m51.343996613s) [upload=464907]: Next returned nil segment
Dec 27 17:34:51.472932614 (530h5m51.344015649s) [upload=464907]: Waiting for error group managing segments...
Dec 27 17:34:51.759416889 (530h5m51.630499924s) [noise=true , address=145.79.227.4:4186    , piece_id=B7L44AGZT2ZCM4WILSZQBCWDR4WASEWMRYNKKIUII4S3NR376PPA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.771187392 (530h5m51.642270427s) [noise=true , address=145.79.227.17:4972   , piece_id=34JJIUEI3MZ6RYPOR7DR4WDRA4HU7T6KCLV3VLMRPDYP6VXODWTQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.774288247 (530h5m51.645371292s) [noise=true , address=145.79.227.14:4839   , piece_id=2NVCYCPU6BQUJT5SAATIITNETMLEQCNSSY5PLDE56RRTUKZSG6UA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.778707291 (530h5m51.649790336s) [noise=true , address=145.79.227.32:4834   , piece_id=QSOOIRQHGTPQLDMDODL2Z3GNBB6M3TNZ5FD5R72MY33YZ6KTWTNA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.779740639 (530h5m51.650823674s) [noise=true , address=145.79.227.50:4375   , piece_id=DX5CF4WG6YZP6CB4K7V6ZK5KYVM4H7KWYXZPGFC4ZEL6OQUDDWQA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.785112077 (530h5m51.656195112s) [noise=true , address=145.79.227.39:4015   , piece_id=UVWN5AX4JDHSZQPJU6JR3XEX7L2XQ2H4DD5A2DNTD6ENQRXTTDOQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.793898699 (530h5m51.664981734s) [noise=true , address=145.79.227.9:4366    , piece_id=6JAH4DWLDO32SKPGCUFO6PP6AFCQFELRQWZ6OIERJWNQ5NXDAW5A, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.794337103 (530h5m51.665420138s) [noise=true , address=145.79.227.52:4447   , piece_id=Q3453I4PXSXFZSOQ7DBJNDFQDKBKQLTM7EF3FQW63P25FUMRCPDQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.800670694 (530h5m51.671753729s) [noise=true , address=145.79.227.10:4403   , piece_id=ZUUGIZ46V4O3F5TGYIGAWTFECYKOUHFBWE27QD2HL2EEXOZUTDWQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:51.813242883 (530h5m51.684325918s) [noise=true , address=145.79.227.48:4301   , piece_id=QGSQ7KUES6GWJ2UOW7UJ43WSLMPDDHQ555X46WSXEPKG3ZANCOEQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:52.041568104 (530h5m51.912651139s) [noise=true , address=45.91.116.7:4011     , piece_id=COPOL5ZS3Q2HAXP4SLWKA5WKAV5FEUZE645S2XTMPIVQ3Z5EUDHQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:52.062767672 (530h5m51.933850717s) [noise=true , address=45.91.116.6:4976     , piece_id=YBJ7QU3C6VE7OVKMHFA2DJ5IMSDDNL26NBBBGSNW4GKZJ4YLO5HQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:52.067920564 (530h5m51.939003639s) [noise=true , address=77.90.31.11:4556     , piece_id=OOANCVFEVLLG3LO2HU4WSPV6VXKEDSCEF3HOZF5PLU5TORYIXDQA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: <nil>
Dec 27 17:34:52.068157645 (530h5m51.93924068s) [seg_pos={0 0}, upload=464907]: Segment reached optimal threshold of 13 pieces.
Dec 27 17:34:52.068482935 (530h5m51.93956601s) [noise=true , address=45.91.116.12:4227    , piece_id=EXCF4SMYHUQGX37EGNJLZF3I7FXMAOEM5SOIZELNHWLH2M2FAM6A, seg_pos={0 0}, upload=464907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 27 17:34:52.068521518 (530h5m51.939604553s) [noise=true , address=45.91.116.12:4227    , piece_id=EXCF4SMYHUQGX37EGNJLZF3I7FXMAOEM5SOIZELNHWLH2M2FAM6A, seg_pos={0 0}, upload=464907]: Long tail cancellation due to optimal threshold
Dec 27 17:34:52.068856105 (530h5m51.93993914s) [noise=true , address=45.91.116.5:4298     , piece_id=QXYGO524EWSMLWVVHYSFRAB445BBMZWST5FKCFFYPDFEZFGESZYQ, seg_pos={0 0}, upload=464907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 17:34:52.068862939 (530h5m51.939945974s) [noise=true , address=45.91.116.5:4298     , piece_id=QXYGO524EWSMLWVVHYSFRAB445BBMZWST5FKCFFYPDFEZFGESZYQ, seg_pos={0 0}, upload=464907]: Long tail cancellation due to optimal threshold
Dec 27 17:34:52.069185092 (530h5m51.940268127s) [noise=true , address=145.79.227.3:4151    , piece_id=7SQKKQTEETYFBG6KF6MF5RL6JP3J454IDF66SBPG42A2QU6AGEOA, seg_pos={0 0}, upload=464907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:59332->145.79.227.3:4151: use of closed network connection; context canceled
Dec 27 17:34:52.069193017 (530h5m51.940276062s) [noise=true , address=145.79.227.3:4151    , piece_id=7SQKKQTEETYFBG6KF6MF5RL6JP3J454IDF66SBPG42A2QU6AGEOA, seg_pos={0 0}, upload=464907]: Long tail cancellation due to optimal threshold
Dec 27 17:34:52.069213807 (530h5m51.940296852s) [seg_pos={0 0}, upload=464907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 17:34:52.069563472 (530h5m51.940646518s) [upload=464907]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 17:34:52.069571157 (530h5m51.940654202s) [upload=464907]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 17:34:52.361027661 (530h5m52.232110706s) [upload=464907]: Done uploading segments...
Dec 27 17:34:52.361051666 (530h5m52.232134701s) [upload=464907]: Upload finished. err: <nil>
Duration: 7.479779824s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [....X.......................................................]
duration: 
3.482304148s
trace: 
https://jaeger.dev.storj.io/trace/63f3b817fe611960
FAIL (2025-12-27 17:35:31 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.482335327s core uplink us-select-1 us1-placement-20