Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 355031 reports (Page 215 of 7101)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [..X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X......]
duration: 
5.972417522s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:59:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 5.972452449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...X...........X....X.X........X................X...........]
duration: 
40.89724796s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:59:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 40.897275062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [...X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.....]
duration: 
5.895534863s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:59:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 5.895557395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
27 out of 60 is failed (threshold=6) [X..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXX]
duration: 
5.000882803s
trace: 
https://jaeger.dev.storj.io/trace/70d07e2c957d00b5
FAIL (2025-12-30 11:59:33 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000903964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
27 out of 60 is failed (threshold=6) [XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXX]
duration: 
5.001385602s
trace: 
https://jaeger.dev.storj.io/trace/45a919a26494f152
FAIL (2025-12-30 11:59:00 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001412964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [X...X...........X....X.X........X................X..........]
duration: 
26.630210313s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:58:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 26.630228588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [....X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X....]
duration: 
6.067044869s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:58:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 6.067072781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.X...X...........X....X.X........X................X.........]
duration: 
32.669828601s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:58:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 32.669846485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..X...X...........X....X.X........X................X........]
duration: 
33.52830405s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:57:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 33.528332263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
26 out of 60 is failed (threshold=6) [.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXX]
duration: 
5.001212843s
trace: 
https://jaeger.dev.storj.io/trace/56f4cb02cadd18b3
FAIL (2025-12-30 11:57:46 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001233963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.....X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X...]
duration: 
5.974232581s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:57:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 5.974260453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...X...X...........X....X.X........X................X.......]
duration: 
33.033444645s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:57:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 33.033465846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
26 out of 60 is failed (threshold=6) [X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXX]
duration: 
5.000950423s
trace: 
https://jaeger.dev.storj.io/trace/1f6e1ad3957ce77a
FAIL (2025-12-30 11:57:10 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000970351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [....X...X...........X....X.X........X................X......]
duration: 
47.602478406s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:56:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 47.60251782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [......X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X..]
duration: 
5.93527637s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:56:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 5.935298051s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXX]
duration: 
5.002357763s
trace: 
https://jaeger.dev.storj.io/trace/3a8e1e5364278f15
FAIL (2025-12-30 11:55:57 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002379474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.......X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X.]
duration: 
5.705821963s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:55:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 5.705849004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.....X...X...........X....X.X........X................X.....]
duration: 
15.449060759s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:55:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 15.449092812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
24 out of 60 is failed (threshold=6) [..X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.XX]
duration: 
5.000652804s
trace: 
https://jaeger.dev.storj.io/trace/5360f3d46e0e209e
FAIL (2025-12-30 11:55:32 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000677951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.X]
duration: 
30.001936391s
trace: 
https://jaeger.dev.storj.io/trace/70e1f6a40dcd4a79
FAIL (2025-12-30 11:55:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6.526556735 (596h25m46.39763978s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:54:46.767868601 (596h25m46.638951646s) [seg_pos={0 0}, upload=522825]: Begin upload segment...
Dec 30 11:54:46.767924687 (596h25m46.639007722s) [seg_pos={0 0}, upload=522825]: Done begin upload segment.
Dec 30 11:54:46.767974733 (596h25m46.639057768s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.767988289 (596h25m46.639071334s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768449557 (596h25m46.639532592s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.768971511 (596h25m46.640054546s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.769655143 (596h25m46.640738178s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770044855 (596h25m46.64112789s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770332804 (596h25m46.641415849s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.770633145 (596h25m46.64171618s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771544420 (596h25m46.642627455s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.771678766 (596h25m46.642761811s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772678710 (596h25m46.643761745s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.772755566 (596h25m46.643838612s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.774138911 (596h25m46.645221946s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.775355978 (596h25m46.646439013s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.776599797 (596h25m46.647682832s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:46.777863233 (596h25m46.648946288s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Uploading piece...
Dec 30 11:54:48.427654402 (596h25m48.298737437s) [upload=522825]: Next returned nil segment
Dec 30 11:54:48.427695560 (596h25m48.298778615s) [upload=522825]: Waiting for error group managing segments...
Dec 30 11:54:48.761968420 (596h25m48.633051455s) [noise=true , address=145.79.227.12:4765   , piece_id=PIOTEWW33IIRBUW2E7JUWKYKLVEAA6RVCOUC4SRTU7HUOWNB27MQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767625531 (596h25m48.638708566s) [noise=true , address=145.79.227.41:4086   , piece_id=3LJ2BORUXTO4IKM34UW5YAOMLER4XPMCOQLKL3AECEQTWUZFSHJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.767778312 (596h25m48.638861347s) [noise=true , address=145.79.227.38:4978   , piece_id=OIPYQ4TQG4HRFS5W2324VI37VYBVLVGXIUC6A2DM4TCTGVB3QS7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.770501847 (596h25m48.641584883s) [noise=true , address=145.79.227.37:4566   , piece_id=JHC3JUZRCLLB5BL3RMRBYW6GZSQF3B3L5V7EMH5CV3BJEWODPL6Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.773729053 (596h25m48.644812098s) [noise=true , address=145.79.227.47:4265   , piece_id=DL2XHNAGYOHW2HQHDXANUNZBME24ZEMW63HFZPMNXGTKOTI7WL7Q, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.774664775 (596h25m48.64574781s) [noise=true , address=145.79.227.9:4365    , piece_id=JSYB6VTZBZ6C3QBPC2IR745B7BBMQ2DHF4N6VY4OJULJ7SO4YQ6A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.776252347 (596h25m48.647335382s) [noise=true , address=145.79.227.5:4221    , piece_id=GRO5BZJDSA2V34Y7G24CTTUNXACI3GPYW4U5E3TOEGZKGTXRBCLA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.788488548 (596h25m48.659571583s) [noise=true , address=145.79.227.4:4187    , piece_id=JPD2B25ZS3PLGYTVS4J52XZ3ZKYHFLNHJJMSUV4KSOCSM455LAXQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.792129291 (596h25m48.663212326s) [noise=true , address=145.79.227.18:4006   , piece_id=3EELAXQ4OL2ML3GGWTGDPK6HU6G3NZRCOKEDHDFGITZ4A3ER2YDA, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.842223678 (596h25m48.713306713s) [noise=true , address=145.79.227.8:4330    , piece_id=5XNFHLWYE7FGYMVJJUU2TTS7SOL2JFVEGSCGXWEN54VKIM4QWIKQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:48.847468915 (596h25m48.71855195s) [noise=true , address=145.79.227.16:4912   , piece_id=MDY2RIGJNJOSQC3YGK732IPKMXCUVKSC3EURTUBLHXUPC4GR4VMQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.094577872 (596h25m48.965660907s) [noise=true , address=45.91.116.14:4191    , piece_id=XGV5N7AYKBTCL3QGA2KX3DCOX26L2FXD2JQUENH6NQ76Y5MSUPPQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462093811 (596h25m49.333176856s) [noise=true , address=45.91.116.13:4155    , piece_id=RKX76Q6CWY4MBKJCODUVUJSSGR5NSUVRVWIQWPQSXNOMOPCPRKVQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: <nil>
Dec 30 11:54:49.462120291 (596h25m49.333203327s) [seg_pos={0 0}, upload=522825]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:54:49.462411405 (596h25m49.33349444s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57078->45.91.116.5:4297: use of closed network connection
Dec 30 11:54:49.462429040 (596h25m49.333512075s) [noise=true , address=45.91.116.5:4297     , piece_id=FXTRBJVVZVNFSQFWYYHSH67TGCPSRAMJ6EJLQZI32REKRDDVQP7A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462539159 (596h25m49.333622204s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40896->45.91.116.4:4937: use of closed network connection
Dec 30 11:54:49.462560039 (596h25m49.333643074s) [noise=true , address=45.91.116.4:4937     , piece_id=XIKPXFN6WVSMLZ2XCNNXCC2ZDP4KOGCW2XF55SQFXPIN7Z3NJPJQ, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462642857 (596h25m49.333725902s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49986->45.91.116.11:4117: use of closed network connection
Dec 30 11:54:49.462658105 (596h25m49.33374115s) [noise=true , address=45.91.116.11:4117    , piece_id=A3GZV5JYG7L2X3JLDEFN5W5XYA6DTLH4ECJOYOC3YPGCF7WAAT3A, seg_pos={0 0}, upload=522825]: Long tail cancellation due to optimal threshold
Dec 30 11:54:49.462681400 (596h25m49.333764435s) [seg_pos={0 0}, upload=522825]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:54:49.462762123 (596h25m49.333845158s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:54:49.462767314 (596h25m49.333850349s) [upload=522825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:54:49.748457192 (596h25m49.619540227s) [upload=522825]: Done uploading segments...
Dec 30 11:54:49.748476609 (596h25m49.619559644s) [upload=522825]: Upload finished. err: <nil>
Duration: 30.001946331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [......X...X...........X....X.X........X................X....]
duration: 
53.395358893s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:55:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 53.395385814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.X]
duration: 
5.001927593s
trace: 
https://jaeger.dev.storj.io/trace/2c6fc516964d59f4
FAIL (2025-12-30 11:54:57 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001951889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X....X.]
duration: 
25.573021167s
trace: 
https://jaeger.dev.storj.io/trace/6d48953c81522599
FAIL (2025-12-30 11:54:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ) context deadline exceeded
======LOGS======
Dec 30 11:53:11.774381296 (596h24m11.645464331s) [upload=522799]: Starting upload
Dec 30 11:53:11.774431982 (596h24m11.645515027s) [upload=522799]: Done starting upload
Dec 30 11:53:11.774583580 (596h24m11.645666615s) [upload=522799]: Uploading segments...
Dec 30 11:53:11.774666899 (596h24m11.645749935s) [upload=522799]: Got next segment. Inline: false
Dec 30 11:53:11.774697307 (596h24m11.645780343s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:53:11.774715642 (596h24m11.645798677s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:53:12.017772992 (596h24m11.888856027s) [seg_pos={0 0}, upload=522799]: Begin upload segment...
Dec 30 11:53:12.017942344 (596h24m11.889025379s) [seg_pos={0 0}, upload=522799]: Done begin upload segment.
Dec 30 11:53:12.017971620 (596h24m11.889054655s) [noise=true , address=145.79.227.4:4187    , piece_id=VVYDYKQU6JLOLDBD6V6DPDIW3T452GCMGC2QLLRHKH7UNIDI354Q, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.018192190 (596h24m11.889275235s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019050925 (596h24m11.890133961s) [noise=true , address=145.79.227.3:4151    , piece_id=WH57XZXVZ2WG4LBMQSFKKDFTR34F46FZP7NHNU6VEH6XDIOJGHYQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019058680 (596h24m11.890141725s) [noise=true , address=145.79.227.42:4123   , piece_id=FPALKP5JE77SYU226LVK5Y72ZUVGNIUHYKXQF7PCHTUFAUEFNSWA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019468070 (596h24m11.890551105s) [noise=true , address=145.79.227.20:4438   , piece_id=DBXQR52OHTFVAJ5EBKDIKLMPV4ZPYS7QFTRYBTSOMEMHPLILQLAQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019670676 (596h24m11.890753711s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019855628 (596h24m11.890938673s) [noise=true , address=45.91.116.14:4191    , piece_id=YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020220773 (596h24m11.891303809s) [noise=true , address=145.79.227.24:4581   , piece_id=6RF5F6ESJVIS7GHNAHACXYZAR4KJF6DFOKSZVFWFZ2S2IAQAHXOQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020425022 (596h24m11.891508057s) [noise=true , address=145.79.227.46:4602   , piece_id=WLPKPAEF75DO6EWBRHJT72ACJKQMIB4YGXHFJL5JGZIU5UW52FNQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020598101 (596h24m11.891681136s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020987183 (596h24m11.892070228s) [noise=true , address=145.79.227.8:4331    , piece_id=5VJK6FUWRO3DPWXXRN5X77RLFFM3VNIYHJVKD4OKQRBW5WEPVOUA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021196421 (596h24m11.892279466s) [noise=true , address=145.79.227.7:4295    , piece_id=NP3CRXMSP5YVV3LVGP2GPBTKWQORV2AKCLUOEGS4OJ4JQLPBQMLQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021360613 (596h24m11.892443648s) [noise=true , address=145.79.227.51:4410   , piece_id=7S3RLTJAQU7X6VPFCOYDPVJLCNFTD7RX7G5DDUDFVOANDWM7PLXA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021900812 (596h24m11.892983857s) [noise=true , address=145.79.227.28:4726   , piece_id=Z5OZSHMKD2RQ2FGFBYI72V4FF7RMKNCXW26YO4YLFMI4KTKXGTQA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.023063416 (596h24m11.894146451s) [noise=true , address=145.79.227.21:4475   , piece_id=J74AWEMIZH2NHYULJVDLXR7DNVU3TCQHDWO63DFQG7P2YVAE3VOA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.025544730 (596h24m11.896627765s) [noise=true , address=45.91.116.6:4975     , piece_id=D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:13.736499123 (596h24m13.607582158s) [upload=522799]: Next returned nil segment
Dec 30 11:53:13.736512509 (596h24m13.607595544s) [upload=522799]: Waiting for error group managing segments...
Dec 30 11:53:14.025121817 (596h24m13.896204852s) [noise=true , address=145.79.227.51:4410   , piece_id=7S3RLTJAQU7X6VPFCOYDPVJLCNFTD7RX7G5DDUDFVOANDWM7PLXA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.025437528 (596h24m13.896520563s) [noise=true , address=145.79.227.42:4123   , piece_id=FPALKP5JE77SYU226LVK5Y72ZUVGNIUHYKXQF7PCHTUFAUEFNSWA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.030473067 (596h24m13.901556102s) [noise=true , address=145.79.227.3:4151    , piece_id=WH57XZXVZ2WG4LBMQSFKKDFTR34F46FZP7NHNU6VEH6XDIOJGHYQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.030604717 (596h24m13.901687762s) [noise=true , address=145.79.227.8:4331    , piece_id=5VJK6FUWRO3DPWXXRN5X77RLFFM3VNIYHJVKD4OKQRBW5WEPVOUA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.038590013 (596h24m13.909673048s) [noise=true , address=145.79.227.46:4602   , piece_id=WLPKPAEF75DO6EWBRHJT72ACJKQMIB4YGXHFJL5JGZIU5UW52FNQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.055848646 (596h24m13.926931691s) [noise=true , address=145.79.227.28:4726   , piece_id=Z5OZSHMKD2RQ2FGFBYI72V4FF7RMKNCXW26YO4YLFMI4KTKXGTQA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.082299683 (596h24m13.953382718s) [noise=true , address=145.79.227.4:4187    , piece_id=VVYDYKQU6JLOLDBD6V6DPDIW3T452GCMGC2QLLRHKH7UNIDI354Q, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.086673382 (596h24m13.957756417s) [noise=true , address=145.79.227.7:4295    , piece_id=NP3CRXMSP5YVV3LVGP2GPBTKWQORV2AKCLUOEGS4OJ4JQLPBQMLQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.091035237 (596h24m13.962118272s) [noise=true , address=145.79.227.24:4581   , piece_id=6RF5F6ESJVIS7GHNAHACXYZAR4KJF6DFOKSZVFWFZ2S2IAQAHXOQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.093671486 (596h24m13.964754531s) [noise=true , address=145.79.227.21:4475   , piece_id=J74AWEMIZH2NHYULJVDLXR7DNVU3TCQHDWO63DFQG7P2YVAE3VOA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.102678778 (596h24m13.973761823s) [noise=true , address=145.79.227.20:4438   , piece_id=DBXQR52OHTFVAJ5EBKDIKLMPV4ZPYS7QFTRYBTSOMEMHPLILQLAQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.328431237 (596h24m14.199514282s) [noise=true , address=45.91.116.14:4191    , piece_id=YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.363670449 (596h24m14.234753494s) [noise=true , address=45.91.116.6:4975     , piece_id=D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.363695255 (596h24m14.23477829s) [seg_pos={0 0}, upload=522799]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:53:14.363892661 (596h24m14.234975696s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.363899564 (596h24m14.234982599s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364065400 (596h24m14.235148435s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.364080789 (596h24m14.235163824s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364100487 (596h24m14.235183522s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.364107620 (596h24m14.235190665s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364125405 (596h24m14.23520844s) [seg_pos={0 0}, upload=522799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:53:14.364219864 (596h24m14.235302899s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:53:14.364227108 (596h24m14.235310144s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:53:14.658011120 (596h24m14.529094155s) [upload=522799]: Done uploading segments...
Dec 30 11:53:14.658030506 (596h24m14.529113541s) [upload=522799]: Upload finished. err: <nil>
Duration: 25.573050103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.......X...X...........X....X.X........X................X...]
duration: 
39.211408599s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:54:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 39.211427065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX.]
duration: 
3.577039507s
trace: 
https://jaeger.dev.storj.io/trace/4868eaba3fe12d53
FAIL (2025-12-30 11:53:43 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.577066819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [..X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X....X]
duration: 
30.004410545s
trace: 
https://jaeger.dev.storj.io/trace/6d48953c81522599
FAIL (2025-12-30 11:53:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ) context deadline exceeded
======LOGS======
Dec 30 11:53:11.774381296 (596h24m11.645464331s) [upload=522799]: Starting upload
Dec 30 11:53:11.774431982 (596h24m11.645515027s) [upload=522799]: Done starting upload
Dec 30 11:53:11.774583580 (596h24m11.645666615s) [upload=522799]: Uploading segments...
Dec 30 11:53:11.774666899 (596h24m11.645749935s) [upload=522799]: Got next segment. Inline: false
Dec 30 11:53:11.774697307 (596h24m11.645780343s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:53:11.774715642 (596h24m11.645798677s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:53:12.017772992 (596h24m11.888856027s) [seg_pos={0 0}, upload=522799]: Begin upload segment...
Dec 30 11:53:12.017942344 (596h24m11.889025379s) [seg_pos={0 0}, upload=522799]: Done begin upload segment.
Dec 30 11:53:12.017971620 (596h24m11.889054655s) [noise=true , address=145.79.227.4:4187    , piece_id=VVYDYKQU6JLOLDBD6V6DPDIW3T452GCMGC2QLLRHKH7UNIDI354Q, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.018192190 (596h24m11.889275235s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019050925 (596h24m11.890133961s) [noise=true , address=145.79.227.3:4151    , piece_id=WH57XZXVZ2WG4LBMQSFKKDFTR34F46FZP7NHNU6VEH6XDIOJGHYQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019058680 (596h24m11.890141725s) [noise=true , address=145.79.227.42:4123   , piece_id=FPALKP5JE77SYU226LVK5Y72ZUVGNIUHYKXQF7PCHTUFAUEFNSWA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019468070 (596h24m11.890551105s) [noise=true , address=145.79.227.20:4438   , piece_id=DBXQR52OHTFVAJ5EBKDIKLMPV4ZPYS7QFTRYBTSOMEMHPLILQLAQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019670676 (596h24m11.890753711s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.019855628 (596h24m11.890938673s) [noise=true , address=45.91.116.14:4191    , piece_id=YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020220773 (596h24m11.891303809s) [noise=true , address=145.79.227.24:4581   , piece_id=6RF5F6ESJVIS7GHNAHACXYZAR4KJF6DFOKSZVFWFZ2S2IAQAHXOQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020425022 (596h24m11.891508057s) [noise=true , address=145.79.227.46:4602   , piece_id=WLPKPAEF75DO6EWBRHJT72ACJKQMIB4YGXHFJL5JGZIU5UW52FNQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020598101 (596h24m11.891681136s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.020987183 (596h24m11.892070228s) [noise=true , address=145.79.227.8:4331    , piece_id=5VJK6FUWRO3DPWXXRN5X77RLFFM3VNIYHJVKD4OKQRBW5WEPVOUA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021196421 (596h24m11.892279466s) [noise=true , address=145.79.227.7:4295    , piece_id=NP3CRXMSP5YVV3LVGP2GPBTKWQORV2AKCLUOEGS4OJ4JQLPBQMLQ, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021360613 (596h24m11.892443648s) [noise=true , address=145.79.227.51:4410   , piece_id=7S3RLTJAQU7X6VPFCOYDPVJLCNFTD7RX7G5DDUDFVOANDWM7PLXA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.021900812 (596h24m11.892983857s) [noise=true , address=145.79.227.28:4726   , piece_id=Z5OZSHMKD2RQ2FGFBYI72V4FF7RMKNCXW26YO4YLFMI4KTKXGTQA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.023063416 (596h24m11.894146451s) [noise=true , address=145.79.227.21:4475   , piece_id=J74AWEMIZH2NHYULJVDLXR7DNVU3TCQHDWO63DFQG7P2YVAE3VOA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:12.025544730 (596h24m11.896627765s) [noise=true , address=45.91.116.6:4975     , piece_id=D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA, seg_pos={0 0}, upload=522799]: Uploading piece...
Dec 30 11:53:13.736499123 (596h24m13.607582158s) [upload=522799]: Next returned nil segment
Dec 30 11:53:13.736512509 (596h24m13.607595544s) [upload=522799]: Waiting for error group managing segments...
Dec 30 11:53:14.025121817 (596h24m13.896204852s) [noise=true , address=145.79.227.51:4410   , piece_id=7S3RLTJAQU7X6VPFCOYDPVJLCNFTD7RX7G5DDUDFVOANDWM7PLXA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.025437528 (596h24m13.896520563s) [noise=true , address=145.79.227.42:4123   , piece_id=FPALKP5JE77SYU226LVK5Y72ZUVGNIUHYKXQF7PCHTUFAUEFNSWA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.030473067 (596h24m13.901556102s) [noise=true , address=145.79.227.3:4151    , piece_id=WH57XZXVZ2WG4LBMQSFKKDFTR34F46FZP7NHNU6VEH6XDIOJGHYQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.030604717 (596h24m13.901687762s) [noise=true , address=145.79.227.8:4331    , piece_id=5VJK6FUWRO3DPWXXRN5X77RLFFM3VNIYHJVKD4OKQRBW5WEPVOUA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.038590013 (596h24m13.909673048s) [noise=true , address=145.79.227.46:4602   , piece_id=WLPKPAEF75DO6EWBRHJT72ACJKQMIB4YGXHFJL5JGZIU5UW52FNQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.055848646 (596h24m13.926931691s) [noise=true , address=145.79.227.28:4726   , piece_id=Z5OZSHMKD2RQ2FGFBYI72V4FF7RMKNCXW26YO4YLFMI4KTKXGTQA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.082299683 (596h24m13.953382718s) [noise=true , address=145.79.227.4:4187    , piece_id=VVYDYKQU6JLOLDBD6V6DPDIW3T452GCMGC2QLLRHKH7UNIDI354Q, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.086673382 (596h24m13.957756417s) [noise=true , address=145.79.227.7:4295    , piece_id=NP3CRXMSP5YVV3LVGP2GPBTKWQORV2AKCLUOEGS4OJ4JQLPBQMLQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.091035237 (596h24m13.962118272s) [noise=true , address=145.79.227.24:4581   , piece_id=6RF5F6ESJVIS7GHNAHACXYZAR4KJF6DFOKSZVFWFZ2S2IAQAHXOQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.093671486 (596h24m13.964754531s) [noise=true , address=145.79.227.21:4475   , piece_id=J74AWEMIZH2NHYULJVDLXR7DNVU3TCQHDWO63DFQG7P2YVAE3VOA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.102678778 (596h24m13.973761823s) [noise=true , address=145.79.227.20:4438   , piece_id=DBXQR52OHTFVAJ5EBKDIKLMPV4ZPYS7QFTRYBTSOMEMHPLILQLAQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.328431237 (596h24m14.199514282s) [noise=true , address=45.91.116.14:4191    , piece_id=YAVWGX6HURIZANG2YUIHVVZP3OJPK4ODJDWJURWEJPUBOCMFUAYQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.363670449 (596h24m14.234753494s) [noise=true , address=45.91.116.6:4975     , piece_id=D4UMEQSALRT7QPXLNLFXLQBCKJZSGGECT3HJWFD6MJGWZQGEZKYA, seg_pos={0 0}, upload=522799]: Done uploading piece. err: <nil>
Dec 30 11:53:14.363695255 (596h24m14.23477829s) [seg_pos={0 0}, upload=522799]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:53:14.363892661 (596h24m14.234975696s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.363899564 (596h24m14.234982599s) [noise=true , address=77.90.31.15:4700     , piece_id=DWPHE7KFKIGDHZSMHLNCLHHL474KP5GPKOTSJAEMHBKCJUU3C7RQ, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364065400 (596h24m14.235148435s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.364080789 (596h24m14.235163824s) [noise=true , address=45.91.116.7:4010     , piece_id=JM2SLHUQAMY7DCMMG5FQNS2UNUNX6GEIISJHFMM2BM6LJAO7QUVQ, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364100487 (596h24m14.235183522s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 11:53:14.364107620 (596h24m14.235190665s) [noise=true , address=45.91.116.8:4048     , piece_id=HF2GDI257PJA7CTLSHGQK6QDIEHWSH3KXVLSIZBOOYQSBIJBL74Q, seg_pos={0 0}, upload=522799]: Long tail cancellation due to optimal threshold
Dec 30 11:53:14.364125405 (596h24m14.23520844s) [seg_pos={0 0}, upload=522799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:53:14.364219864 (596h24m14.235302899s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:53:14.364227108 (596h24m14.235310144s) [upload=522799]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:53:14.658011120 (596h24m14.529094155s) [upload=522799]: Done uploading segments...
Dec 30 11:53:14.658030506 (596h24m14.529113541s) [upload=522799]: Upload finished. err: <nil>
Duration: 30.004428801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [........X...X...........X....X.X........X................X..]
duration: 
48.060619561s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:53:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 48.060636784s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....XX]
duration: 
5.001685694s
trace: 
https://jaeger.dev.storj.io/trace/4868eaba3fe12d53
FAIL (2025-12-30 11:53:10 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001717685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [...X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X....]
duration: 
5.797089008s
trace: 
https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426
FAIL (2025-12-30 11:52:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload
Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload
Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments...
Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false
Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment...
Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment.
Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment
Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments...
Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection
Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments...
Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil>
Duration: 5.79712697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.........X...X...........X....X.X........X................X.]
duration: 
43.254156697s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:52:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 43.254172476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....X]
duration: 
5.001629664s
trace: 
https://jaeger.dev.storj.io/trace/786e0102014f212f
FAIL (2025-12-30 11:52:07 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001651818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X...]
duration: 
6.011147371s
trace: 
https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426
FAIL (2025-12-30 11:51:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload
Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload
Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments...
Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false
Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment...
Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment.
Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment
Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments...
Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection
Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments...
Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil>
Duration: 6.011228215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..........X...X...........X....X.X........X................X]
duration: 
1m0.003743641s
trace: 
https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134
FAIL (2025-12-30 11:51:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
1m57.317198375s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece...
Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000
Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment
Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments...
Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124    , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211    , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023   , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824   , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897     , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717     , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865     , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532     , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170    , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196   , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925   , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771   , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899   , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964     , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil>
Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection
Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838     , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection
Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715    , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF
Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671     , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold
Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments...
Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil>
Duration: 1m0.003767184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X..]
duration: 
26.836734833s
trace: 
https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426
FAIL (2025-12-30 11:51:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload
Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload
Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments...
Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false
Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment...
Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment.
Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment
Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments...
Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection
Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments...
Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil>
Duration: 26.836758428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
22 out of 60 is failed (threshold=6) [.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....]
duration: 
4.725355215s
trace: 
https://jaeger.dev.storj.io/trace/6e904e080c2bce20
FAIL (2025-12-30 11:51:01 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.725387085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X.]
duration: 
6.655171003s
trace: 
https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426
FAIL (2025-12-30 11:50:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload
Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload
Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments...
Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false
Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment...
Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment.
Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment
Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments...
Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection
Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments...
Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil>
Duration: 6.655192244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [..X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X]
duration: 
30.003760995s
trace: 
https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426
FAIL (2025-12-30 11:50:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload
Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload
Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments...
Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false
Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment...
Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment.
Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece...
Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment
Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments...
Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729   , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403   , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565   , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439   , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015   , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259    , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375   , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475   , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601   , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662     , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937     , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155    , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009     , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil>
Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection
Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226    , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975     , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339   , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold
Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments...
Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil>
Duration: 30.003786253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [X..........X...X...........X....X.X........X................]
duration: 
31.93811298s
trace: 
https://jaeger.dev.storj.io/trace/40a87f21171f227
FAIL (2025-12-30 11:50:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, upload=522543]: Uploading piece...
Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment
Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments...
Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775   , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000
Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection
Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments...
Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil>
Duration: 31.93812705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.X..........X...X...........X....X.X........X...............]
duration: 
52.91699797s
trace: 
https://jaeger.dev.storj.io/trace/40a87f21171f227
FAIL (2025-12-30 11:49:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, upload=522543]: Uploading piece...
Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment
Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments...
Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775   , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000
Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection
Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments...
Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil>
Duration: 52.917024391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X...]
duration: 
4.241886092s
trace: 
https://jaeger.dev.storj.io/trace/6e904e080c2bce20
FAIL (2025-12-30 11:49:49 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.24192722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [...X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.]
duration: 
5.748341836s
trace: 
https://jaeger.dev.storj.io/trace/56ff714953efb52f
FAIL (2025-12-30 11:49:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 11:48:21.820684079 (596h19m21.691767124s) [upload=522727]: Starting upload
Dec 30 11:48:21.820790401 (596h19m21.691873466s) [upload=522727]: Done starting upload
Dec 30 11:48:21.820908907 (596h19m21.691991952s) [upload=522727]: Uploading segments...
Dec 30 11:48:21.821039355 (596h19m21.6921224s) [upload=522727]: Got next segment. Inline: false
Dec 30 11:48:21.821086915 (596h19m21.69216995s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:48:21.821093338 (596h19m21.692176374s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:48:22.064127253 (596h19m21.935210319s) [seg_pos={0 0}, upload=522727]: Begin upload segment...
Dec 30 11:48:22.064185515 (596h19m21.93526855s) [seg_pos={0 0}, upload=522727]: Done begin upload segment.
Dec 30 11:48:22.064217906 (596h19m21.935300982s) [noise=true , address=145.79.227.48:4303   , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.064274914 (596h19m21.935357959s) [noise=true , address=145.79.227.16:4910   , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.065018511 (596h19m21.936101566s) [noise=true , address=77.90.31.11:4555     , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.065678107 (596h19m21.936761142s) [noise=true , address=145.79.227.26:4653   , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.066625571 (596h19m21.937708606s) [noise=true , address=145.79.227.7:4294    , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067091147 (596h19m21.938174182s) [noise=true , address=145.79.227.17:4970   , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067389496 (596h19m21.938472541s) [noise=true , address=145.79.227.47:4266   , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067478334 (596h19m21.938561369s) [noise=true , address=45.91.116.13:4153    , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067859089 (596h19m21.938942134s) [noise=true , address=145.79.227.14:4839   , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068025116 (596h19m21.939108161s) [noise=true , address=145.79.227.5:4221    , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068285190 (596h19m21.939368235s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068674903 (596h19m21.939757948s) [noise=true , address=45.91.116.12:4225    , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068682748 (596h19m21.939765783s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.069094162 (596h19m21.940177197s) [noise=true , address=145.79.227.11:4730   , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.069282400 (596h19m21.940365445s) [noise=true , address=145.79.227.40:4051   , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.070105487 (596h19m21.941188522s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:23.775774806 (596h19m23.646857841s) [upload=522727]: Next returned nil segment
Dec 30 11:48:23.775787771 (596h19m23.646870816s) [upload=522727]: Waiting for error group managing segments...
Dec 30 11:48:24.070297073 (596h19m23.941380118s) [noise=true , address=145.79.227.5:4221    , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.073745420 (596h19m23.944828455s) [noise=true , address=145.79.227.17:4970   , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.074159528 (596h19m23.945242573s) [noise=true , address=145.79.227.11:4730   , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.074564790 (596h19m23.945647835s) [noise=true , address=145.79.227.7:4294    , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.078996298 (596h19m23.950079343s) [noise=true , address=145.79.227.26:4653   , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.082836511 (596h19m23.953919546s) [noise=true , address=145.79.227.16:4910   , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.135298577 (596h19m24.006381622s) [noise=true , address=145.79.227.14:4839   , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.140443945 (596h19m24.01152699s) [noise=true , address=145.79.227.47:4266   , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.226240736 (596h19m24.097323781s) [noise=true , address=145.79.227.48:4303   , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.483097262 (596h19m24.354180307s) [noise=true , address=45.91.116.13:4153    , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.526902163 (596h19m24.397985219s) [noise=true , address=45.91.116.12:4225    , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.537354927 (596h19m24.408437962s) [noise=true , address=77.90.31.11:4555     , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:25.444901482 (596h19m25.315984527s) [noise=true , address=145.79.227.40:4051   , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:25.444920918 (596h19m25.316003953s) [seg_pos={0 0}, upload=522727]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:48:25.445122342 (596h19m25.316205417s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445131970 (596h19m25.316215005s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445296674 (596h19m25.316379709s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445302935 (596h19m25.316385981s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445362118 (596h19m25.316445164s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445372708 (596h19m25.316455753s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445415390 (596h19m25.316498435s) [seg_pos={0 0}, upload=522727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:48:25.445546339 (596h19m25.316629384s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:48:25.445567920 (596h19m25.316650955s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:48:25.733181612 (596h19m25.604264657s) [upload=522727]: Done uploading segments...
Dec 30 11:48:25.733205869 (596h19m25.604288914s) [upload=522727]: Upload finished. err: <nil>
Duration: 5.748383515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
23 out of 60 is failed (threshold=6) [.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X..]
duration: 
5.002177741s
trace: 
https://jaeger.dev.storj.io/trace/6e904e080c2bce20
FAIL (2025-12-30 11:48:59 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.002204392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..X..........X...X...........X....X.X........X..............]
duration: 
48.985987393s
trace: 
https://jaeger.dev.storj.io/trace/40a87f21171f227
FAIL (2025-12-30 11:48:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, upload=522543]: Uploading piece...
Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment
Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments...
Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775   , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000
Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection
Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments...
Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil>
Duration: 48.986026689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [....X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X]
duration: 
30.003216271s
trace: 
https://jaeger.dev.storj.io/trace/56ff714953efb52f
FAIL (2025-12-30 11:48:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 11:48:21.820684079 (596h19m21.691767124s) [upload=522727]: Starting upload
Dec 30 11:48:21.820790401 (596h19m21.691873466s) [upload=522727]: Done starting upload
Dec 30 11:48:21.820908907 (596h19m21.691991952s) [upload=522727]: Uploading segments...
Dec 30 11:48:21.821039355 (596h19m21.6921224s) [upload=522727]: Got next segment. Inline: false
Dec 30 11:48:21.821086915 (596h19m21.69216995s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:48:21.821093338 (596h19m21.692176374s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:48:22.064127253 (596h19m21.935210319s) [seg_pos={0 0}, upload=522727]: Begin upload segment...
Dec 30 11:48:22.064185515 (596h19m21.93526855s) [seg_pos={0 0}, upload=522727]: Done begin upload segment.
Dec 30 11:48:22.064217906 (596h19m21.935300982s) [noise=true , address=145.79.227.48:4303   , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.064274914 (596h19m21.935357959s) [noise=true , address=145.79.227.16:4910   , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.065018511 (596h19m21.936101566s) [noise=true , address=77.90.31.11:4555     , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.065678107 (596h19m21.936761142s) [noise=true , address=145.79.227.26:4653   , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.066625571 (596h19m21.937708606s) [noise=true , address=145.79.227.7:4294    , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067091147 (596h19m21.938174182s) [noise=true , address=145.79.227.17:4970   , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067389496 (596h19m21.938472541s) [noise=true , address=145.79.227.47:4266   , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067478334 (596h19m21.938561369s) [noise=true , address=45.91.116.13:4153    , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.067859089 (596h19m21.938942134s) [noise=true , address=145.79.227.14:4839   , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068025116 (596h19m21.939108161s) [noise=true , address=145.79.227.5:4221    , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068285190 (596h19m21.939368235s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068674903 (596h19m21.939757948s) [noise=true , address=45.91.116.12:4225    , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.068682748 (596h19m21.939765783s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.069094162 (596h19m21.940177197s) [noise=true , address=145.79.227.11:4730   , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.069282400 (596h19m21.940365445s) [noise=true , address=145.79.227.40:4051   , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:22.070105487 (596h19m21.941188522s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Uploading piece...
Dec 30 11:48:23.775774806 (596h19m23.646857841s) [upload=522727]: Next returned nil segment
Dec 30 11:48:23.775787771 (596h19m23.646870816s) [upload=522727]: Waiting for error group managing segments...
Dec 30 11:48:24.070297073 (596h19m23.941380118s) [noise=true , address=145.79.227.5:4221    , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.073745420 (596h19m23.944828455s) [noise=true , address=145.79.227.17:4970   , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.074159528 (596h19m23.945242573s) [noise=true , address=145.79.227.11:4730   , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.074564790 (596h19m23.945647835s) [noise=true , address=145.79.227.7:4294    , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.078996298 (596h19m23.950079343s) [noise=true , address=145.79.227.26:4653   , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.082836511 (596h19m23.953919546s) [noise=true , address=145.79.227.16:4910   , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.135298577 (596h19m24.006381622s) [noise=true , address=145.79.227.14:4839   , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.140443945 (596h19m24.01152699s) [noise=true , address=145.79.227.47:4266   , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.226240736 (596h19m24.097323781s) [noise=true , address=145.79.227.48:4303   , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.483097262 (596h19m24.354180307s) [noise=true , address=45.91.116.13:4153    , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.526902163 (596h19m24.397985219s) [noise=true , address=45.91.116.12:4225    , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:24.537354927 (596h19m24.408437962s) [noise=true , address=77.90.31.11:4555     , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:25.444901482 (596h19m25.315984527s) [noise=true , address=145.79.227.40:4051   , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil>
Dec 30 11:48:25.444920918 (596h19m25.316003953s) [seg_pos={0 0}, upload=522727]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:48:25.445122342 (596h19m25.316205417s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445131970 (596h19m25.316215005s) [noise=true , address=145.79.227.33:4871   , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445296674 (596h19m25.316379709s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445302935 (596h19m25.316385981s) [noise=true , address=77.90.31.15:4700     , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445362118 (596h19m25.316445164s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 30 11:48:25.445372708 (596h19m25.316455753s) [noise=true , address=145.79.227.49:4338   , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold
Dec 30 11:48:25.445415390 (596h19m25.316498435s) [seg_pos={0 0}, upload=522727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:48:25.445546339 (596h19m25.316629384s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:48:25.445567920 (596h19m25.316650955s) [upload=522727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:48:25.733181612 (596h19m25.604264657s) [upload=522727]: Done uploading segments...
Dec 30 11:48:25.733205869 (596h19m25.604288914s) [upload=522727]: Upload finished. err: <nil>
Duration: 30.003319597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [.....X....X.......X.....X..XXX....XX.X.X.X........X...X..X..]
duration: 
5.875487851s
trace: 
https://jaeger.dev.storj.io/trace/37bbc15678dcb08d
FAIL (2025-12-30 11:47:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ) context deadline exceeded
======LOGS======
Dec 30 11:46:16.208132621 (596h17m16.079215657s) [upload=522695]: Starting upload
Dec 30 11:46:16.208186665 (596h17m16.07926971s) [upload=522695]: Done starting upload
Dec 30 11:46:16.208323715 (596h17m16.07940675s) [upload=522695]: Uploading segments...
Dec 30 11:46:16.208437532 (596h17m16.079520567s) [upload=522695]: Got next segment. Inline: false
Dec 30 11:46:16.208509760 (596h17m16.079592795s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:46:16.208515220 (596h17m16.079598255s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:46:16.454687231 (596h17m16.325770266s) [seg_pos={0 0}, upload=522695]: Begin upload segment...
Dec 30 11:46:16.454753057 (596h17m16.325836092s) [seg_pos={0 0}, upload=522695]: Done begin upload segment.
Dec 30 11:46:16.454796579 (596h17m16.325879614s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.454909324 (596h17m16.325992369s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.455692065 (596h17m16.32677516s) [noise=true , address=145.79.227.16:4912   , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456031170 (596h17m16.327114215s) [noise=true , address=145.79.227.5:4221    , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456265256 (596h17m16.327348301s) [noise=true , address=145.79.227.11:4730   , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456370787 (596h17m16.327453822s) [noise=true , address=145.79.227.20:4439   , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456642865 (596h17m16.3277259s) [noise=true , address=145.79.227.21:4473   , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456819411 (596h17m16.327902446s) [noise=true , address=145.79.227.47:4267   , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457042716 (596h17m16.328125751s) [noise=true , address=145.79.227.23:4547   , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457341996 (596h17m16.328425031s) [noise=true , address=145.79.227.32:4835   , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457495288 (596h17m16.328578333s) [noise=true , address=77.90.31.19:4842     , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457613854 (596h17m16.328696899s) [noise=true , address=145.79.227.51:4410   , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458139254 (596h17m16.329222299s) [noise=true , address=45.91.116.4:4940     , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458155886 (596h17m16.329238931s) [noise=true , address=145.79.227.36:4517   , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458789002 (596h17m16.329872047s) [noise=true , address=77.90.31.18:4805     , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.459435062 (596h17m16.330518107s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:18.064524931 (596h17m17.935607966s) [upload=522695]: Next returned nil segment
Dec 30 11:46:18.064543617 (596h17m17.935626662s) [upload=522695]: Waiting for error group managing segments...
Dec 30 11:46:18.445650235 (596h17m18.31673327s) [noise=true , address=145.79.227.21:4473   , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.445674992 (596h17m18.316758037s) [noise=true , address=145.79.227.51:4410   , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.446391677 (596h17m18.317474712s) [noise=true , address=145.79.227.16:4912   , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.447931178 (596h17m18.319014224s) [noise=true , address=145.79.227.32:4835   , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.450052899 (596h17m18.321135944s) [noise=true , address=145.79.227.20:4439   , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.458643858 (596h17m18.329726903s) [noise=true , address=145.79.227.11:4730   , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.462219195 (596h17m18.33330224s) [noise=true , address=145.79.227.5:4221    , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.463608330 (596h17m18.334691365s) [noise=true , address=145.79.227.36:4517   , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.473171460 (596h17m18.344254495s) [noise=true , address=145.79.227.23:4547   , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.474467548 (596h17m18.345550583s) [noise=true , address=145.79.227.47:4267   , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.732587350 (596h17m18.603670395s) [noise=true , address=77.90.31.18:4805     , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.746855699 (596h17m18.617938744s) [noise=true , address=77.90.31.19:4842     , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.975180716 (596h17m18.846263752s) [noise=true , address=45.91.116.4:4940     , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.975208249 (596h17m18.846291294s) [seg_pos={0 0}, upload=522695]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:46:18.975470548 (596h17m18.846553613s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 30 11:46:18.975480316 (596h17m18.846563361s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975756262 (596h17m18.846839307s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:43450->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 11:46:18.975782592 (596h17m18.846865657s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975843028 (596h17m18.846926083s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 30 11:46:18.975855421 (596h17m18.846938466s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975883404 (596h17m18.846966439s) [seg_pos={0 0}, upload=522695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:46:18.975997271 (596h17m18.847080326s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:46:18.976005808 (596h17m18.847088853s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:46:19.262987707 (596h17m19.134070752s) [upload=522695]: Done uploading segments...
Dec 30 11:46:19.263009128 (596h17m19.134092163s) [upload=522695]: Upload finished. err: <nil>
Duration: 5.875518168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
24 out of 60 is failed (threshold=6) [X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X.]
duration: 
3.462359518s
trace: 
https://jaeger.dev.storj.io/trace/6e904e080c2bce20
FAIL (2025-12-30 11:47:47 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.462476801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...X..........X...X...........X....X.X........X.............]
duration: 
51.759723491s
trace: 
https://jaeger.dev.storj.io/trace/40a87f21171f227
FAIL (2025-12-30 11:47:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, upload=522543]: Uploading piece...
Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece...
Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment
Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments...
Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283    , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495   , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169   , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583     , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868     , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319     , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528     , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800     , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775   , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963     , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000
Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067   , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898   , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225   , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931   , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350   , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil>
Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces.
Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513   , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection
Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715    , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF
Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold
Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments...
Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil>
Duration: 51.759752186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
24 out of 60 is failed (threshold=6) [.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X]
duration: 
5.001559201s
trace: 
https://jaeger.dev.storj.io/trace/6e904e080c2bce20
FAIL (2025-12-30 11:47:31 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001581864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
24 out of 60 is failed (threshold=6) [X.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...]
duration: 
5.001583737s
trace: 
https://jaeger.dev.storj.io/trace/17445fb928c372b5
FAIL (2025-12-30 11:47:09 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001660183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [......X....X.......X.....X..XXX....XX.X.X.X........X...X..X.]
duration: 
6.010534724s
trace: 
https://jaeger.dev.storj.io/trace/37bbc15678dcb08d
FAIL (2025-12-30 11:47:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ) context deadline exceeded
======LOGS======
Dec 30 11:46:16.208132621 (596h17m16.079215657s) [upload=522695]: Starting upload
Dec 30 11:46:16.208186665 (596h17m16.07926971s) [upload=522695]: Done starting upload
Dec 30 11:46:16.208323715 (596h17m16.07940675s) [upload=522695]: Uploading segments...
Dec 30 11:46:16.208437532 (596h17m16.079520567s) [upload=522695]: Got next segment. Inline: false
Dec 30 11:46:16.208509760 (596h17m16.079592795s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 11:46:16.208515220 (596h17m16.079598255s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 11:46:16.454687231 (596h17m16.325770266s) [seg_pos={0 0}, upload=522695]: Begin upload segment...
Dec 30 11:46:16.454753057 (596h17m16.325836092s) [seg_pos={0 0}, upload=522695]: Done begin upload segment.
Dec 30 11:46:16.454796579 (596h17m16.325879614s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.454909324 (596h17m16.325992369s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.455692065 (596h17m16.32677516s) [noise=true , address=145.79.227.16:4912   , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456031170 (596h17m16.327114215s) [noise=true , address=145.79.227.5:4221    , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456265256 (596h17m16.327348301s) [noise=true , address=145.79.227.11:4730   , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456370787 (596h17m16.327453822s) [noise=true , address=145.79.227.20:4439   , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456642865 (596h17m16.3277259s) [noise=true , address=145.79.227.21:4473   , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.456819411 (596h17m16.327902446s) [noise=true , address=145.79.227.47:4267   , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457042716 (596h17m16.328125751s) [noise=true , address=145.79.227.23:4547   , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457341996 (596h17m16.328425031s) [noise=true , address=145.79.227.32:4835   , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457495288 (596h17m16.328578333s) [noise=true , address=77.90.31.19:4842     , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.457613854 (596h17m16.328696899s) [noise=true , address=145.79.227.51:4410   , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458139254 (596h17m16.329222299s) [noise=true , address=45.91.116.4:4940     , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458155886 (596h17m16.329238931s) [noise=true , address=145.79.227.36:4517   , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.458789002 (596h17m16.329872047s) [noise=true , address=77.90.31.18:4805     , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:16.459435062 (596h17m16.330518107s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Uploading piece...
Dec 30 11:46:18.064524931 (596h17m17.935607966s) [upload=522695]: Next returned nil segment
Dec 30 11:46:18.064543617 (596h17m17.935626662s) [upload=522695]: Waiting for error group managing segments...
Dec 30 11:46:18.445650235 (596h17m18.31673327s) [noise=true , address=145.79.227.21:4473   , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.445674992 (596h17m18.316758037s) [noise=true , address=145.79.227.51:4410   , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.446391677 (596h17m18.317474712s) [noise=true , address=145.79.227.16:4912   , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.447931178 (596h17m18.319014224s) [noise=true , address=145.79.227.32:4835   , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.450052899 (596h17m18.321135944s) [noise=true , address=145.79.227.20:4439   , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.458643858 (596h17m18.329726903s) [noise=true , address=145.79.227.11:4730   , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.462219195 (596h17m18.33330224s) [noise=true , address=145.79.227.5:4221    , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.463608330 (596h17m18.334691365s) [noise=true , address=145.79.227.36:4517   , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.473171460 (596h17m18.344254495s) [noise=true , address=145.79.227.23:4547   , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.474467548 (596h17m18.345550583s) [noise=true , address=145.79.227.47:4267   , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.732587350 (596h17m18.603670395s) [noise=true , address=77.90.31.18:4805     , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.746855699 (596h17m18.617938744s) [noise=true , address=77.90.31.19:4842     , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.975180716 (596h17m18.846263752s) [noise=true , address=45.91.116.4:4940     , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil>
Dec 30 11:46:18.975208249 (596h17m18.846291294s) [seg_pos={0 0}, upload=522695]: Segment reached optimal threshold of 13 pieces.
Dec 30 11:46:18.975470548 (596h17m18.846553613s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 30 11:46:18.975480316 (596h17m18.846563361s) [noise=true , address=45.91.116.8:4047     , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975756262 (596h17m18.846839307s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:43450->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 11:46:18.975782592 (596h17m18.846865657s) [noise=true , address=45.91.116.6:4976     , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975843028 (596h17m18.846926083s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 30 11:46:18.975855421 (596h17m18.846938466s) [noise=true , address=45.91.116.10:4084    , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold
Dec 30 11:46:18.975883404 (596h17m18.846966439s) [seg_pos={0 0}, upload=522695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 11:46:18.975997271 (596h17m18.847080326s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 11:46:18.976005808 (596h17m18.847088853s) [upload=522695]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 11:46:19.262987707 (596h17m19.134070752s) [upload=522695]: Done uploading segments...
Dec 30 11:46:19.263009128 (596h17m19.134092163s) [upload=522695]: Upload finished. err: <nil>
Duration: 6.010562827s core uplink us-select-1