Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X] duration: 30.002593996s trace: https://jaeger.dev.storj.io/trace/114f5caeb7488ee0 FAIL (2025-12-27 03:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VEO5NLJRGOZGHWFZDJSLRGL3ZY4OYYGE6QY56BXJCQXJ42KSZA4Q) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: FAGW7PGARZFEFNXTMLO6FUAL7SAEWEHMLJEHCJL7I5QT6ZVEGCTQ) context deadline exceeded ======LOGS====== [upload=452384]: Uploading segments... Dec 27 03:04:08.913013951 (515h35m8.784096996s) [upload=452384]: Got next segment. Inline: false Dec 27 03:04:08.913055109 (515h35m8.784138144s) [upload=452384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:04:08.913059537 (515h35m8.784142582s) [upload=452384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:04:09.158691073 (515h35m9.029774108s) [seg_pos={0 0}, upload=452384]: Begin upload segment... Dec 27 03:04:09.158759223 (515h35m9.029842258s) [seg_pos={0 0}, upload=452384]: Done begin upload segment. Dec 27 03:04:09.158802795 (515h35m9.02988583s) [noise=true , address=45.91.116.10:4083 , piece_id=GSY2QPZB5URDLXCB5DOOMHTRAX7HO4GMFATIEAZFTXGBTR36CZBA, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.159267751 (515h35m9.030350786s) [noise=true , address=45.91.116.5:4297 , piece_id=VEO5NLJRGOZGHWFZDJSLRGL3ZY4OYYGE6QY56BXJCQXJ42KSZA4Q, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.159982722 (515h35m9.031065757s) [noise=true , address=145.79.227.48:4302 , piece_id=VLRGOG7X2BJP2FJL2MI7XMS2GDUUHDJNHHRW4GSPFJNN24LSWCMQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.160405427 (515h35m9.031488462s) [noise=true , address=145.79.227.5:4221 , piece_id=NASYGAPSMIXRFKWWIQA5RIKYUH3EHRKKM6OS27A2JXVZMEXBRZCQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.160875182 (515h35m9.031958227s) [noise=true , address=145.79.227.45:4231 , piece_id=YMFXNJZ7E4C56ZQBW4GL4OOECUUPV7MESYFC37MCX57JECO2QNYA, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.161027081 (515h35m9.032110126s) [noise=true , address=145.79.227.38:4978 , piece_id=2LBW7MI6JG2JZX5ZWZHG7RSHAIDO3L2SYPYO4AIYPVAJQCAY2B4Q, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.161407115 (515h35m9.03249016s) [noise=true , address=45.91.116.6:4973 , piece_id=FAGW7PGARZFEFNXTMLO6FUAL7SAEWEHMLJEHCJL7I5QT6ZVEGCTQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.161770877 (515h35m9.032853922s) [noise=true , address=77.90.31.12:4589 , piece_id=TREQ2MFA7U37XIE2EVBRRCAXET7NML36MYSYAD6TMCUDEDJRQUPQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.162194665 (515h35m9.03327771s) [noise=true , address=145.79.227.52:4445 , piece_id=IBLPTKCST4H34LVMQHJX5PYS7QLTMAAE3PWLJGZGYUJL6LUXYCFA, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.162626026 (515h35m9.033709061s) [noise=true , address=77.90.31.17:4773 , piece_id=XDTT7NW4DYCO4UWYCMXIT2RPQ73A7AP4RN5YQWXUSRQYQGP5Z6DQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.163036277 (515h35m9.034119322s) [noise=true , address=45.91.116.12:4228 , piece_id=KTUHZ3PED4TTTF6K7MBLPWUTZ67LORUYTCGWQEEEPFEPPX6BS3SQ, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.163418244 (515h35m9.034501279s) [noise=true , address=145.79.227.37:4567 , piece_id=PVJ646OPLM4FTRTHODLSNGNRD2Z3RD6FGDHEPR5NSSYWRC5EXP6A, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.163793068 (515h35m9.034876103s) [noise=true , address=45.91.116.13:4153 , piece_id=YMPK2PJJML7QIZGMCZIFQMM7HS4PG5IWRLMKB2NDBE3JULNP274A, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.174297431 (515h35m9.045380466s) [noise=true , address=45.91.116.4:4937 , piece_id=43WNP2ZGA3HKQSCA5PF2NHPWIUZHDLSUF6KL4CGJSVN43EDRNTJA, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.177497765 (515h35m9.0485808s) [noise=true , address=145.79.227.15:4873 , piece_id=FJQGN5WEX3ISGQ2KD5ZUBKMOZ5WK65EIT2WZ2ZO4OY35NW5HP47A, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:09.179700769 (515h35m9.050783814s) [noise=true , address=45.91.116.11:4118 , piece_id=QB754FWWX2R7JGA3CT5TIUQUN62QNBI47FGI7JJISJBIXHSGYTXA, seg_pos={0 0}, upload=452384]: Uploading piece... Dec 27 03:04:10.808839170 (515h35m10.679922205s) [upload=452384]: Next returned nil segment Dec 27 03:04:10.808851985 (515h35m10.67993502s) [upload=452384]: Waiting for error group managing segments... Dec 27 03:04:11.169714906 (515h35m11.040797941s) [noise=true , address=145.79.227.45:4231 , piece_id=YMFXNJZ7E4C56ZQBW4GL4OOECUUPV7MESYFC37MCX57JECO2QNYA, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.173627497 (515h35m11.044710532s) [noise=true , address=145.79.227.15:4873 , piece_id=FJQGN5WEX3ISGQ2KD5ZUBKMOZ5WK65EIT2WZ2ZO4OY35NW5HP47A, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.177493007 (515h35m11.048576052s) [noise=true , address=145.79.227.52:4445 , piece_id=IBLPTKCST4H34LVMQHJX5PYS7QLTMAAE3PWLJGZGYUJL6LUXYCFA, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.189189610 (515h35m11.060272645s) [noise=true , address=145.79.227.37:4567 , piece_id=PVJ646OPLM4FTRTHODLSNGNRD2Z3RD6FGDHEPR5NSSYWRC5EXP6A, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.194953866 (515h35m11.066036901s) [noise=true , address=145.79.227.48:4302 , piece_id=VLRGOG7X2BJP2FJL2MI7XMS2GDUUHDJNHHRW4GSPFJNN24LSWCMQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.203972710 (515h35m11.075055755s) [noise=true , address=145.79.227.38:4978 , piece_id=2LBW7MI6JG2JZX5ZWZHG7RSHAIDO3L2SYPYO4AIYPVAJQCAY2B4Q, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.218063703 (515h35m11.089146748s) [noise=true , address=145.79.227.5:4221 , piece_id=NASYGAPSMIXRFKWWIQA5RIKYUH3EHRKKM6OS27A2JXVZMEXBRZCQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.391422805 (515h35m11.26250585s) [noise=true , address=45.91.116.12:4228 , piece_id=KTUHZ3PED4TTTF6K7MBLPWUTZ67LORUYTCGWQEEEPFEPPX6BS3SQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.406433729 (515h35m11.277516764s) [noise=true , address=45.91.116.6:4973 , piece_id=FAGW7PGARZFEFNXTMLO6FUAL7SAEWEHMLJEHCJL7I5QT6ZVEGCTQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.414877448 (515h35m11.285960493s) [noise=true , address=45.91.116.4:4937 , piece_id=43WNP2ZGA3HKQSCA5PF2NHPWIUZHDLSUF6KL4CGJSVN43EDRNTJA, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.426945678 (515h35m11.298028733s) [noise=true , address=45.91.116.11:4118 , piece_id=QB754FWWX2R7JGA3CT5TIUQUN62QNBI47FGI7JJISJBIXHSGYTXA, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.474917797 (515h35m11.346000853s) [noise=true , address=45.91.116.13:4153 , piece_id=YMPK2PJJML7QIZGMCZIFQMM7HS4PG5IWRLMKB2NDBE3JULNP274A, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.586152763 (515h35m11.457235808s) [noise=true , address=45.91.116.5:4297 , piece_id=VEO5NLJRGOZGHWFZDJSLRGL3ZY4OYYGE6QY56BXJCQXJ42KSZA4Q, seg_pos={0 0}, upload=452384]: Done uploading piece. err: <nil> Dec 27 03:04:11.586221213 (515h35m11.457304258s) [seg_pos={0 0}, upload=452384]: Segment reached optimal threshold of 13 pieces. Dec 27 03:04:11.586421545 (515h35m11.4575046s) [noise=true , address=77.90.31.17:4773 , piece_id=XDTT7NW4DYCO4UWYCMXIT2RPQ73A7AP4RN5YQWXUSRQYQGP5Z6DQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:34900->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:34900->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34900->77.90.31.17:4773: use of closed network connection Dec 27 03:04:11.586429470 (515h35m11.457512515s) [noise=true , address=77.90.31.17:4773 , piece_id=XDTT7NW4DYCO4UWYCMXIT2RPQ73A7AP4RN5YQWXUSRQYQGP5Z6DQ, seg_pos={0 0}, upload=452384]: Long tail cancellation due to optimal threshold Dec 27 03:04:11.586797350 (515h35m11.457880396s) [noise=true , address=77.90.31.12:4589 , piece_id=TREQ2MFA7U37XIE2EVBRRCAXET7NML36MYSYAD6TMCUDEDJRQUPQ, seg_pos={0 0}, upload=452384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:48992->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:48992->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48992->77.90.31.12:4589: use of closed network connection Dec 27 03:04:11.586808091 (515h35m11.457891136s) [noise=true , address=77.90.31.12:4589 , piece_id=TREQ2MFA7U37XIE2EVBRRCAXET7NML36MYSYAD6TMCUDEDJRQUPQ, seg_pos={0 0}, upload=452384]: Long tail cancellation due to optimal threshold Dec 27 03:04:11.586849970 (515h35m11.457933015s) [noise=true , address=45.91.116.10:4083 , piece_id=GSY2QPZB5URDLXCB5DOOMHTRAX7HO4GMFATIEAZFTXGBTR36CZBA, seg_pos={0 0}, upload=452384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 03:04:11.586869678 (515h35m11.457952743s) [noise=true , address=45.91.116.10:4083 , piece_id=GSY2QPZB5URDLXCB5DOOMHTRAX7HO4GMFATIEAZFTXGBTR36CZBA, seg_pos={0 0}, upload=452384]: Long tail cancellation due to optimal threshold Dec 27 03:04:11.586917318 (515h35m11.458000364s) [seg_pos={0 0}, upload=452384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:04:11.587020054 (515h35m11.458103099s) [upload=452384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:04:11.587027548 (515h35m11.458110593s) [upload=452384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:04:11.882674946 (515h35m11.753757981s) [upload=452384]: Done uploading segments... Dec 27 03:04:11.882712417 (515h35m11.753795462s) [upload=452384]: Upload finished. err: <nil> Duration: 30.002615828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 30.017057812s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:03:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 30.017074545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX...] duration: 6.120590691s trace: https://jaeger.dev.storj.io/trace/5a9616351db06f96 FAIL (2025-12-27 03:03:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 03:01:27.751977661 (515h32m27.623060706s) [upload=452346]: Starting upload Dec 27 03:01:27.752061801 (515h32m27.623144846s) [upload=452346]: Done starting upload Dec 27 03:01:27.752169065 (515h32m27.62325211s) [upload=452346]: Uploading segments... Dec 27 03:01:27.752314522 (515h32m27.623397567s) [upload=452346]: Got next segment. Inline: false Dec 27 03:01:27.752361873 (515h32m27.623444908s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:01:27.752368054 (515h32m27.623451089s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:01:27.995611211 (515h32m27.866694266s) [seg_pos={0 0}, upload=452346]: Begin upload segment... Dec 27 03:01:27.995664211 (515h32m27.866747256s) [seg_pos={0 0}, upload=452346]: Done begin upload segment. Dec 27 03:01:27.995693757 (515h32m27.866776803s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995701573 (515h32m27.866784618s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995827482 (515h32m27.866910517s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996230089 (515h32m27.867313124s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996642224 (515h32m27.867725269s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997045892 (515h32m27.868128937s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997448930 (515h32m27.868531975s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997920187 (515h32m27.869003222s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998228224 (515h32m27.869311259s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998472960 (515h32m27.869556005s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998708689 (515h32m27.869791734s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999078704 (515h32m27.870161739s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999522198 (515h32m27.870605234s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000553091 (515h32m27.871636136s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000747031 (515h32m27.871830096s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.001331614 (515h32m27.872414669s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:29.616573132 (515h32m29.487656167s) [upload=452346]: Next returned nil segment Dec 27 03:01:29.616585906 (515h32m29.487668951s) [upload=452346]: Waiting for error group managing segments... Dec 27 03:01:30.002675945 (515h32m29.87375898s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.002815010 (515h32m29.873898045s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.003352372 (515h32m29.874435407s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.014813297 (515h32m29.885896332s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.016102652 (515h32m29.887185687s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.018229061 (515h32m29.889312096s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.021724918 (515h32m29.892807953s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.022442804 (515h32m29.893525849s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.182952329 (515h32m30.054035374s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.237608055 (515h32m30.10869109s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.249157728 (515h32m30.120240773s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.258828133 (515h32m30.129911168s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291856296 (515h32m30.162939341s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291878218 (515h32m30.162961253s) [seg_pos={0 0}, upload=452346]: Segment reached optimal threshold of 13 pieces. Dec 27 03:01:30.292109237 (515h32m30.163192272s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection Dec 27 03:01:30.292116701 (515h32m30.163199746s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292320850 (515h32m30.163403915s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292344285 (515h32m30.16342733s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292362590 (515h32m30.163445645s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292371537 (515h32m30.163454582s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292395853 (515h32m30.163478888s) [seg_pos={0 0}, upload=452346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:01:30.292486025 (515h32m30.16356906s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:01:30.292492166 (515h32m30.163575211s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:01:30.578313051 (515h32m30.449396086s) [upload=452346]: Done uploading segments... Dec 27 03:01:30.578332708 (515h32m30.449415753s) [upload=452346]: Upload finished. err: <nil> Duration: 6.12062655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 38.155832779s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 38.155850001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX..] duration: 19.93392303s trace: https://jaeger.dev.storj.io/trace/5a9616351db06f96 FAIL (2025-12-27 03:03:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 03:01:27.751977661 (515h32m27.623060706s) [upload=452346]: Starting upload Dec 27 03:01:27.752061801 (515h32m27.623144846s) [upload=452346]: Done starting upload Dec 27 03:01:27.752169065 (515h32m27.62325211s) [upload=452346]: Uploading segments... Dec 27 03:01:27.752314522 (515h32m27.623397567s) [upload=452346]: Got next segment. Inline: false Dec 27 03:01:27.752361873 (515h32m27.623444908s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:01:27.752368054 (515h32m27.623451089s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:01:27.995611211 (515h32m27.866694266s) [seg_pos={0 0}, upload=452346]: Begin upload segment... Dec 27 03:01:27.995664211 (515h32m27.866747256s) [seg_pos={0 0}, upload=452346]: Done begin upload segment. Dec 27 03:01:27.995693757 (515h32m27.866776803s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995701573 (515h32m27.866784618s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995827482 (515h32m27.866910517s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996230089 (515h32m27.867313124s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996642224 (515h32m27.867725269s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997045892 (515h32m27.868128937s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997448930 (515h32m27.868531975s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997920187 (515h32m27.869003222s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998228224 (515h32m27.869311259s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998472960 (515h32m27.869556005s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998708689 (515h32m27.869791734s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999078704 (515h32m27.870161739s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999522198 (515h32m27.870605234s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000553091 (515h32m27.871636136s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000747031 (515h32m27.871830096s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.001331614 (515h32m27.872414669s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:29.616573132 (515h32m29.487656167s) [upload=452346]: Next returned nil segment Dec 27 03:01:29.616585906 (515h32m29.487668951s) [upload=452346]: Waiting for error group managing segments... Dec 27 03:01:30.002675945 (515h32m29.87375898s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.002815010 (515h32m29.873898045s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.003352372 (515h32m29.874435407s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.014813297 (515h32m29.885896332s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.016102652 (515h32m29.887185687s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.018229061 (515h32m29.889312096s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.021724918 (515h32m29.892807953s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.022442804 (515h32m29.893525849s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.182952329 (515h32m30.054035374s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.237608055 (515h32m30.10869109s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.249157728 (515h32m30.120240773s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.258828133 (515h32m30.129911168s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291856296 (515h32m30.162939341s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291878218 (515h32m30.162961253s) [seg_pos={0 0}, upload=452346]: Segment reached optimal threshold of 13 pieces. Dec 27 03:01:30.292109237 (515h32m30.163192272s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection Dec 27 03:01:30.292116701 (515h32m30.163199746s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292320850 (515h32m30.163403915s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292344285 (515h32m30.16342733s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292362590 (515h32m30.163445645s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292371537 (515h32m30.163454582s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292395853 (515h32m30.163478888s) [seg_pos={0 0}, upload=452346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:01:30.292486025 (515h32m30.16356906s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:01:30.292492166 (515h32m30.163575211s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:01:30.578313051 (515h32m30.449396086s) [upload=452346]: Done uploading segments... Dec 27 03:01:30.578332708 (515h32m30.449415753s) [upload=452346]: Upload finished. err: <nil> Duration: 19.93394446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX.] duration: 25.039583291s trace: https://jaeger.dev.storj.io/trace/5a9616351db06f96 FAIL (2025-12-27 03:02:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 03:01:27.751977661 (515h32m27.623060706s) [upload=452346]: Starting upload Dec 27 03:01:27.752061801 (515h32m27.623144846s) [upload=452346]: Done starting upload Dec 27 03:01:27.752169065 (515h32m27.62325211s) [upload=452346]: Uploading segments... Dec 27 03:01:27.752314522 (515h32m27.623397567s) [upload=452346]: Got next segment. Inline: false Dec 27 03:01:27.752361873 (515h32m27.623444908s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:01:27.752368054 (515h32m27.623451089s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:01:27.995611211 (515h32m27.866694266s) [seg_pos={0 0}, upload=452346]: Begin upload segment... Dec 27 03:01:27.995664211 (515h32m27.866747256s) [seg_pos={0 0}, upload=452346]: Done begin upload segment. Dec 27 03:01:27.995693757 (515h32m27.866776803s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995701573 (515h32m27.866784618s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995827482 (515h32m27.866910517s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996230089 (515h32m27.867313124s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996642224 (515h32m27.867725269s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997045892 (515h32m27.868128937s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997448930 (515h32m27.868531975s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997920187 (515h32m27.869003222s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998228224 (515h32m27.869311259s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998472960 (515h32m27.869556005s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998708689 (515h32m27.869791734s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999078704 (515h32m27.870161739s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999522198 (515h32m27.870605234s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000553091 (515h32m27.871636136s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000747031 (515h32m27.871830096s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.001331614 (515h32m27.872414669s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:29.616573132 (515h32m29.487656167s) [upload=452346]: Next returned nil segment Dec 27 03:01:29.616585906 (515h32m29.487668951s) [upload=452346]: Waiting for error group managing segments... Dec 27 03:01:30.002675945 (515h32m29.87375898s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.002815010 (515h32m29.873898045s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.003352372 (515h32m29.874435407s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.014813297 (515h32m29.885896332s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.016102652 (515h32m29.887185687s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.018229061 (515h32m29.889312096s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.021724918 (515h32m29.892807953s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.022442804 (515h32m29.893525849s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.182952329 (515h32m30.054035374s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.237608055 (515h32m30.10869109s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.249157728 (515h32m30.120240773s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.258828133 (515h32m30.129911168s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291856296 (515h32m30.162939341s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291878218 (515h32m30.162961253s) [seg_pos={0 0}, upload=452346]: Segment reached optimal threshold of 13 pieces. Dec 27 03:01:30.292109237 (515h32m30.163192272s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection Dec 27 03:01:30.292116701 (515h32m30.163199746s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292320850 (515h32m30.163403915s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292344285 (515h32m30.16342733s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292362590 (515h32m30.163445645s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292371537 (515h32m30.163454582s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292395853 (515h32m30.163478888s) [seg_pos={0 0}, upload=452346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:01:30.292486025 (515h32m30.16356906s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:01:30.292492166 (515h32m30.163575211s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:01:30.578313051 (515h32m30.449396086s) [upload=452346]: Done uploading segments... Dec 27 03:01:30.578332708 (515h32m30.449415753s) [upload=452346]: Upload finished. err: <nil> Duration: 25.039610732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 32.328116319s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:02:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 32.328150064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX] duration: 30.003992146s trace: https://jaeger.dev.storj.io/trace/5a9616351db06f96 FAIL (2025-12-27 03:01:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 03:01:27.751977661 (515h32m27.623060706s) [upload=452346]: Starting upload Dec 27 03:01:27.752061801 (515h32m27.623144846s) [upload=452346]: Done starting upload Dec 27 03:01:27.752169065 (515h32m27.62325211s) [upload=452346]: Uploading segments... Dec 27 03:01:27.752314522 (515h32m27.623397567s) [upload=452346]: Got next segment. Inline: false Dec 27 03:01:27.752361873 (515h32m27.623444908s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:01:27.752368054 (515h32m27.623451089s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:01:27.995611211 (515h32m27.866694266s) [seg_pos={0 0}, upload=452346]: Begin upload segment... Dec 27 03:01:27.995664211 (515h32m27.866747256s) [seg_pos={0 0}, upload=452346]: Done begin upload segment. Dec 27 03:01:27.995693757 (515h32m27.866776803s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995701573 (515h32m27.866784618s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.995827482 (515h32m27.866910517s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996230089 (515h32m27.867313124s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.996642224 (515h32m27.867725269s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997045892 (515h32m27.868128937s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997448930 (515h32m27.868531975s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.997920187 (515h32m27.869003222s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998228224 (515h32m27.869311259s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998472960 (515h32m27.869556005s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.998708689 (515h32m27.869791734s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999078704 (515h32m27.870161739s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:27.999522198 (515h32m27.870605234s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000553091 (515h32m27.871636136s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.000747031 (515h32m27.871830096s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:28.001331614 (515h32m27.872414669s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Uploading piece... Dec 27 03:01:29.616573132 (515h32m29.487656167s) [upload=452346]: Next returned nil segment Dec 27 03:01:29.616585906 (515h32m29.487668951s) [upload=452346]: Waiting for error group managing segments... Dec 27 03:01:30.002675945 (515h32m29.87375898s) [noise=true , address=145.79.227.36:4518 , piece_id=XQOOYYZSWSSAHG6U4VGNRP6YUQE6SUYYHTLD53T7HX5Q7A5W3EMQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.002815010 (515h32m29.873898045s) [noise=true , address=145.79.227.13:4803 , piece_id=RTHZMQ6ARXFX6FMXZFCL7KOOYYDJEA5E2DKM3KHXLOECNYZCY46A, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.003352372 (515h32m29.874435407s) [noise=true , address=145.79.227.52:4445 , piece_id=MP4SO5ULLGZCQXSOYT3LHBT4O4QZLGNK4YOX4SUEYL5YRX3IGIIQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.014813297 (515h32m29.885896332s) [noise=true , address=145.79.227.5:4221 , piece_id=VGL5PRTOND2GNWTVOTCSFYXIYVAMRLBADJHYWJTEBI3U3YLAFRDQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.016102652 (515h32m29.887185687s) [noise=true , address=145.79.227.51:4411 , piece_id=3ZZT27X7ITWK7ZLUGQA3B2OMVBQSRCKINUPLKWEBP77KXFLYIUXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.018229061 (515h32m29.889312096s) [noise=true , address=145.79.227.48:4303 , piece_id=4RDBK57TJWA5345IVBVO2TULOY4FQY4MCM3PQR4YM4UB455E6RKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.021724918 (515h32m29.892807953s) [noise=true , address=145.79.227.39:4013 , piece_id=JVOH7OWKPKYUXKVAF4KJ447SI2MNAJLNCTYXVLGTDPBJPK5T7GPA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.022442804 (515h32m29.893525849s) [noise=true , address=145.79.227.17:4971 , piece_id=WEEAWUDXBRCKZJVPEM4AK3L2D5VE4VWFAROGWBA3MXIBBU4JHOKA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.182952329 (515h32m30.054035374s) [noise=true , address=77.90.31.12:4589 , piece_id=GZLXHBCBV37HJKLRGTVLRKLDNWGCZQYIMX2DFN7DKFDV7DTFCWAA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.237608055 (515h32m30.10869109s) [noise=true , address=45.91.116.6:4973 , piece_id=3GXIXVNB5EUC3AXYTIL5KESUGAFCAJZKM26KNPLBLYKIH3TQDLHA, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.249157728 (515h32m30.120240773s) [noise=true , address=77.90.31.17:4770 , piece_id=NN4IOHTOAKHGEW3X3Y3XI2H2RD72SB37FRY6AQGHJRGT2KM6QSXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.258828133 (515h32m30.129911168s) [noise=true , address=45.91.116.13:4155 , piece_id=6EHIYVAVH2P6CV6X37ODKADX2EVZQ5RZ43OH5WGL67WPZF2P5S2Q, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291856296 (515h32m30.162939341s) [noise=true , address=77.90.31.11:4554 , piece_id=IUBJX7NYEZOC5QRVOC5CTWBCTNRMUFKPQMQTQ4CMUTYFDW2YTCLQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: <nil> Dec 27 03:01:30.291878218 (515h32m30.162961253s) [seg_pos={0 0}, upload=452346]: Segment reached optimal threshold of 13 pieces. Dec 27 03:01:30.292109237 (515h32m30.163192272s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58098->45.91.116.4:4940: use of closed network connection Dec 27 03:01:30.292116701 (515h32m30.163199746s) [noise=true , address=45.91.116.4:4940 , piece_id=VA4K2NGKMKNHY5RECNUTKGIZI2YQT56EJTCSM3VJNS6BHI73DFJQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292320850 (515h32m30.163403915s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292344285 (515h32m30.16342733s) [noise=true , address=77.90.31.19:4844 , piece_id=4RPAE62CAZ3YJBAFDQHGYG2UE2ISQI6FQLV3JK776FEMRLWYOJXQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292362590 (515h32m30.163445645s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 03:01:30.292371537 (515h32m30.163454582s) [noise=true , address=145.79.227.44:4195 , piece_id=FBBCSWLGZIQHBXTG72ZKPL5D5BBHWNKWVLRNHCVY3DOIDXWMTQQQ, seg_pos={0 0}, upload=452346]: Long tail cancellation due to optimal threshold Dec 27 03:01:30.292395853 (515h32m30.163478888s) [seg_pos={0 0}, upload=452346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:01:30.292486025 (515h32m30.16356906s) [upload=452346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:01:30.292492166 (515h32m30.163575211s) [upload=452346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:01:30.578313051 (515h32m30.449396086s) [upload=452346]: Done uploading segments... Dec 27 03:01:30.578332708 (515h32m30.449415753s) [upload=452346]: Upload finished. err: <nil> Duration: 30.004013205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 28.107359438s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:01:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 28.107377932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XXX] duration: 30.004301154s trace: https://jaeger.dev.storj.io/trace/6158c34edc01dd89 FAIL (2025-12-27 03:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:00:49.129492773 (515h31m49.000575818s) [upload=452339]: Starting upload Dec 27 03:00:49.129562826 (515h31m49.000645861s) [upload=452339]: Done starting upload Dec 27 03:00:49.129828702 (515h31m49.000911747s) [upload=452339]: Uploading segments... Dec 27 03:00:49.129873768 (515h31m49.000956813s) [upload=452339]: Got next segment. Inline: false Dec 27 03:00:49.129928923 (515h31m49.001011968s) [upload=452339]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:00:49.129934433 (515h31m49.001017478s) [upload=452339]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:00:49.379267026 (515h31m49.250350061s) [seg_pos={0 0}, upload=452339]: Begin upload segment... Dec 27 03:00:49.379326249 (515h31m49.250409294s) [seg_pos={0 0}, upload=452339]: Done begin upload segment. Dec 27 03:00:49.379357328 (515h31m49.250440363s) [noise=true , address=145.79.227.38:4978 , piece_id=UCPLZ3IC3AX3JBNCWVJIMQSTNHJFC2SL2JQRROOZEQDYR3TS3RCA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.379481664 (515h31m49.250564719s) [noise=true , address=145.79.227.44:4194 , piece_id=UNY2GOJWQ2VYSE4LH2LIAJKQ4CWIDIX2KU6ROE5ON2TIHJTI4GIQ, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.380613209 (515h31m49.251696244s) [noise=true , address=77.90.31.19:4841 , piece_id=CR6SPR3KH7TYVMPJMQI3RMFHNNGGI5ZJ4667IUD2ZB225UKSRKCA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.380871962 (515h31m49.251955007s) [noise=true , address=145.79.227.17:4971 , piece_id=WLMKCJSJTL4W6LXDVVWRFOE2N44RMVZ2KB66N3JOGJ5H2IYM4NCA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.381906702 (515h31m49.252989737s) [noise=true , address=145.79.227.48:4302 , piece_id=GGPI2DOZYRFADHNJBITHI7MP4AJLHQGAJ6BXY2YMOQMNRCPDPU5A, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.383025633 (515h31m49.254108678s) [noise=true , address=77.90.31.17:4773 , piece_id=6JF22CBSLD3WC25OBMDZTT33OZKCXWYENDT4HEC6POF6LBBSEO4A, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.383401288 (515h31m49.254484323s) [noise=true , address=145.79.227.47:4265 , piece_id=BARNDUZTMDPZDDT4NHACFZDHT7RMKALCL2W2JT5QQK2ADNBKCU3A, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.384673481 (515h31m49.255756526s) [noise=true , address=145.79.227.36:4517 , piece_id=NTR2Y6Q3R3TTIP4DQFCM6Z3LSWNA65X2R6OIYB3X2JLEJOX2JOPA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.386005507 (515h31m49.257088542s) [noise=true , address=45.91.116.6:4976 , piece_id=6CHTQR6O34E6YJBRACKFA7LEI7WGQ5UQ2P3IYSHZ3I4DWBQ5PJTQ, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.386678449 (515h31m49.257761494s) [noise=true , address=45.91.116.9:4264 , piece_id=EFGTPJGFJT2BV4RBO2V6VFHW5V5MNWYN67HWYSYF6HRODAXPIHGA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.388003993 (515h31m49.259087038s) [noise=true , address=145.79.227.5:4222 , piece_id=S6XF2FQYDSMM7LER3PYFOCTUCVIW3HRZJWM4AQO6B3FHVVUQDMNA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.389023615 (515h31m49.26010665s) [noise=true , address=145.79.227.11:4731 , piece_id=HLUDIUGMRDTNOB65OP7RNVG2EZ6P2HNOFNCH7HQQ77DYEHHHKPEA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.389236430 (515h31m49.260319475s) [noise=true , address=145.79.227.37:4567 , piece_id=RGVLHQYMUCGEQUHA4NRCXQCVQAMOZ7LWKVI4CWOXJX6S35BOFCTA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.389910955 (515h31m49.260994s) [noise=true , address=45.91.116.14:4189 , piece_id=B3VXLTBWRFQKHVD4YLUDHHL3KP2TNRDP6BAEMW62B24E6TYTZAKA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.390291249 (515h31m49.261374284s) [noise=true , address=145.79.227.33:4871 , piece_id=QQUC7K526WJXN5TG35JVGIR2OGA7PIBPEGFTVUIVTPJI2RFLGGDQ, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:49.390763438 (515h31m49.261846473s) [noise=true , address=77.90.31.21:4878 , piece_id=CUQZVEH2TMSKW6GWQTBOFSOZLT3Q2QCOW2E57SNGC53LS67LXPRA, seg_pos={0 0}, upload=452339]: Uploading piece... Dec 27 03:00:51.099004687 (515h31m50.970087732s) [upload=452339]: Next returned nil segment Dec 27 03:00:51.099017842 (515h31m50.970100887s) [upload=452339]: Waiting for error group managing segments... Dec 27 03:00:51.379407404 (515h31m51.25049045s) [noise=true , address=145.79.227.17:4971 , piece_id=WLMKCJSJTL4W6LXDVVWRFOE2N44RMVZ2KB66N3JOGJ5H2IYM4NCA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.396721404 (515h31m51.267804449s) [noise=true , address=145.79.227.11:4731 , piece_id=HLUDIUGMRDTNOB65OP7RNVG2EZ6P2HNOFNCH7HQQ77DYEHHHKPEA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.408632465 (515h31m51.279715511s) [noise=true , address=145.79.227.48:4302 , piece_id=GGPI2DOZYRFADHNJBITHI7MP4AJLHQGAJ6BXY2YMOQMNRCPDPU5A, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.410177597 (515h31m51.281260632s) [noise=true , address=145.79.227.5:4222 , piece_id=S6XF2FQYDSMM7LER3PYFOCTUCVIW3HRZJWM4AQO6B3FHVVUQDMNA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.415783493 (515h31m51.286866528s) [noise=true , address=145.79.227.38:4978 , piece_id=UCPLZ3IC3AX3JBNCWVJIMQSTNHJFC2SL2JQRROOZEQDYR3TS3RCA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.416375761 (515h31m51.287458796s) [noise=true , address=145.79.227.36:4517 , piece_id=NTR2Y6Q3R3TTIP4DQFCM6Z3LSWNA65X2R6OIYB3X2JLEJOX2JOPA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.439117556 (515h31m51.310200601s) [noise=true , address=145.79.227.47:4265 , piece_id=BARNDUZTMDPZDDT4NHACFZDHT7RMKALCL2W2JT5QQK2ADNBKCU3A, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.451323699 (515h31m51.322406744s) [noise=true , address=145.79.227.37:4567 , piece_id=RGVLHQYMUCGEQUHA4NRCXQCVQAMOZ7LWKVI4CWOXJX6S35BOFCTA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.646236795 (515h31m51.51731983s) [noise=true , address=45.91.116.9:4264 , piece_id=EFGTPJGFJT2BV4RBO2V6VFHW5V5MNWYN67HWYSYF6HRODAXPIHGA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.676058552 (515h31m51.547141597s) [noise=true , address=77.90.31.17:4773 , piece_id=6JF22CBSLD3WC25OBMDZTT33OZKCXWYENDT4HEC6POF6LBBSEO4A, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.688364795 (515h31m51.55944784s) [noise=true , address=45.91.116.14:4189 , piece_id=B3VXLTBWRFQKHVD4YLUDHHL3KP2TNRDP6BAEMW62B24E6TYTZAKA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.697141999 (515h31m51.568225035s) [noise=true , address=77.90.31.19:4841 , piece_id=CR6SPR3KH7TYVMPJMQI3RMFHNNGGI5ZJ4667IUD2ZB225UKSRKCA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.833163363 (515h31m51.704246398s) [noise=true , address=45.91.116.6:4976 , piece_id=6CHTQR6O34E6YJBRACKFA7LEI7WGQ5UQ2P3IYSHZ3I4DWBQ5PJTQ, seg_pos={0 0}, upload=452339]: Done uploading piece. err: <nil> Dec 27 03:00:51.833184534 (515h31m51.704267579s) [seg_pos={0 0}, upload=452339]: Segment reached optimal threshold of 13 pieces. Dec 27 03:00:51.833383021 (515h31m51.704466056s) [noise=true , address=145.79.227.44:4194 , piece_id=UNY2GOJWQ2VYSE4LH2LIAJKQ4CWIDIX2KU6ROE5ON2TIHJTI4GIQ, seg_pos={0 0}, upload=452339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 03:00:51.833391518 (515h31m51.704474553s) [noise=true , address=145.79.227.44:4194 , piece_id=UNY2GOJWQ2VYSE4LH2LIAJKQ4CWIDIX2KU6ROE5ON2TIHJTI4GIQ, seg_pos={0 0}, upload=452339]: Long tail cancellation due to optimal threshold Dec 27 03:00:51.833628119 (515h31m51.704711154s) [noise=true , address=77.90.31.21:4878 , piece_id=CUQZVEH2TMSKW6GWQTBOFSOZLT3Q2QCOW2E57SNGC53LS67LXPRA, seg_pos={0 0}, upload=452339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:46762->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:46762->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46762->77.90.31.21:4878: use of closed network connection Dec 27 03:00:51.833635613 (515h31m51.704718648s) [noise=true , address=77.90.31.21:4878 , piece_id=CUQZVEH2TMSKW6GWQTBOFSOZLT3Q2QCOW2E57SNGC53LS67LXPRA, seg_pos={0 0}, upload=452339]: Long tail cancellation due to optimal threshold Dec 27 03:00:51.833905778 (515h31m51.704988813s) [noise=true , address=145.79.227.33:4871 , piece_id=QQUC7K526WJXN5TG35JVGIR2OGA7PIBPEGFTVUIVTPJI2RFLGGDQ, seg_pos={0 0}, upload=452339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 03:00:51.833928149 (515h31m51.705011195s) [noise=true , address=145.79.227.33:4871 , piece_id=QQUC7K526WJXN5TG35JVGIR2OGA7PIBPEGFTVUIVTPJI2RFLGGDQ, seg_pos={0 0}, upload=452339]: Long tail cancellation due to optimal threshold Dec 27 03:00:51.833964619 (515h31m51.705047664s) [seg_pos={0 0}, upload=452339]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:00:51.834072123 (515h31m51.705155189s) [upload=452339]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:00:51.834080731 (515h31m51.705163776s) [upload=452339]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:00:52.121944135 (515h31m51.9930272s) [upload=452339]: Done uploading segments... Dec 27 03:00:52.121968360 (515h31m51.993051395s) [upload=452339]: Upload finished. err: <nil> Duration: 30.00432528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 39.270129669s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:01:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 39.270143675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 24.747359133s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:00:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 24.747389252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.XX] duration: 30.006073069s trace: https://jaeger.dev.storj.io/trace/2eb65ff83b425f83 FAIL (2025-12-27 03:00:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: QIN3KJ4KEEZF5GOSVKNFKX6D45DQDRMLLAUBHXDIW2D7MO45LOWQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: AZNIH3BVMIHI6AGMLYHELREOCOF7XZXGY5BZU653QQN5OZUKJ2MA) piecestore close: write tcp 188.166.225.170:41684->45.91.116.5:4298: use of closed network connection; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: NCZNYXTLJ47Z2QBDBQ3DUZH3GK7MLCIRYRR7OTI7PNEWZ3TOT6XQ) context deadline exceeded ======LOGS====== h30m34.458284346s) [upload=452318]: Done starting upload Dec 27 02:59:34.587282546 (515h30m34.458365581s) [upload=452318]: Uploading segments... Dec 27 02:59:34.587372998 (515h30m34.458456033s) [upload=452318]: Got next segment. Inline: false Dec 27 02:59:34.587401402 (515h30m34.458484437s) [upload=452318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:59:34.587405460 (515h30m34.458488495s) [upload=452318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:59:34.832682781 (515h30m34.703765836s) [seg_pos={0 0}, upload=452318]: Begin upload segment... Dec 27 02:59:34.832733467 (515h30m34.703816502s) [seg_pos={0 0}, upload=452318]: Done begin upload segment. Dec 27 02:59:34.832751682 (515h30m34.703834738s) [noise=true , address=145.79.227.42:4121 , piece_id=VTXMZEOD3CERZHAOSBZBUXP7PUODGAICI55JOYQC7ITGPBG7UREQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.832765158 (515h30m34.703848193s) [noise=true , address=45.91.116.14:4189 , piece_id=Q6CO4VRZE7QQGC7O4WUPUMFPWTGNTL6ZX6CHZZRJ4VM6DNDAXYRA, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.833662827 (515h30m34.704745872s) [noise=true , address=45.91.116.8:4048 , piece_id=EYUG6K4FJIKZLXUIBRKBF4RTAQQRZG3VHTTK6NRQKVOQGRZC5ZFA, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.834019707 (515h30m34.705102742s) [noise=true , address=45.91.116.11:4118 , piece_id=QIN3KJ4KEEZF5GOSVKNFKX6D45DQDRMLLAUBHXDIW2D7MO45LOWQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.834270514 (515h30m34.705353549s) [noise=true , address=145.79.227.16:4912 , piece_id=MSZJWTMKLH3NHR3D62P6ZUP5DQPCSWVFCFG3A6LKEAYLLVR34Y7Q, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.834591276 (515h30m34.705674321s) [noise=true , address=45.91.116.5:4298 , piece_id=AZNIH3BVMIHI6AGMLYHELREOCOF7XZXGY5BZU653QQN5OZUKJ2MA, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.834841291 (515h30m34.705924326s) [noise=true , address=45.91.116.7:4012 , piece_id=XN3GANEWJQ6ABXY2VP5ZKECIPPMHXXUOOOWNYN443XKGNRKRZNYQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.835084665 (515h30m34.7061677s) [noise=true , address=145.79.227.51:4411 , piece_id=VRZ6YJSMTWWBPOQZO5LQTBMAKTJXIVUYHJV6MW477T67GU7XRIDQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.835325193 (515h30m34.706408228s) [noise=true , address=145.79.227.39:4013 , piece_id=IGFHVLKGL55LPXB7ZXGZEMJ24NXXWIZDMYEXQVS6CW4W5Z36SG6A, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.835567625 (515h30m34.70665066s) [noise=true , address=145.79.227.7:4293 , piece_id=NCZNYXTLJ47Z2QBDBQ3DUZH3GK7MLCIRYRR7OTI7PNEWZ3TOT6XQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.835802803 (515h30m34.706885838s) [noise=true , address=45.91.116.4:4939 , piece_id=VP6YHPMIPHTHZRF7WO3YKKQJ4FBPD4BDAUD6YD2ZMFQIW7V2FMQQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.836048180 (515h30m34.707131215s) [noise=true , address=145.79.227.48:4302 , piece_id=FLRQ7LOU6SQK64F3NSGW4S5VEXCEOEP32ZX4DGM65QTKTLWGTRSA, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.836303736 (515h30m34.707386771s) [noise=true , address=145.79.227.38:4978 , piece_id=6YG2IB6C3TQICPB4KJTTHKQT3X7X43IWUE6DZ6VOIOE2RVAF4U2A, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.836544625 (515h30m34.70762766s) [noise=true , address=145.79.227.5:4222 , piece_id=VGS73ZMHOSLNRB4VJURJ4AFCPGB3OTGHCEIBZKRUGXACJ7JBKN7A, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.836803507 (515h30m34.707886543s) [noise=true , address=145.79.227.33:4871 , piece_id=UI75MILQ4RUOGSLSGEQ3TYVBFSDS7BDSA5SC6QVNG7MTHKDS2SDQ, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:34.838029532 (515h30m34.709112577s) [noise=true , address=145.79.227.37:4565 , piece_id=SI7XBCFMP7LBNDWVP6RZUYDWVZN4WZ7SQ3X5UQYNKQEBTPBH7T6Q, seg_pos={0 0}, upload=452318]: Uploading piece... Dec 27 02:59:36.549562290 (515h30m36.420645325s) [upload=452318]: Next returned nil segment Dec 27 02:59:36.549597687 (515h30m36.420680732s) [upload=452318]: Waiting for error group managing segments... Dec 27 02:59:36.841750409 (515h30m36.712833444s) [noise=true , address=145.79.227.51:4411 , piece_id=VRZ6YJSMTWWBPOQZO5LQTBMAKTJXIVUYHJV6MW477T67GU7XRIDQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.849442836 (515h30m36.720525871s) [noise=true , address=145.79.227.38:4978 , piece_id=6YG2IB6C3TQICPB4KJTTHKQT3X7X43IWUE6DZ6VOIOE2RVAF4U2A, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.850203886 (515h30m36.721286931s) [noise=true , address=145.79.227.7:4293 , piece_id=NCZNYXTLJ47Z2QBDBQ3DUZH3GK7MLCIRYRR7OTI7PNEWZ3TOT6XQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.851023256 (515h30m36.722106291s) [noise=true , address=145.79.227.42:4121 , piece_id=VTXMZEOD3CERZHAOSBZBUXP7PUODGAICI55JOYQC7ITGPBG7UREQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.858418839 (515h30m36.729501884s) [noise=true , address=145.79.227.37:4565 , piece_id=SI7XBCFMP7LBNDWVP6RZUYDWVZN4WZ7SQ3X5UQYNKQEBTPBH7T6Q, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.858766041 (515h30m36.729849076s) [noise=true , address=145.79.227.48:4302 , piece_id=FLRQ7LOU6SQK64F3NSGW4S5VEXCEOEP32ZX4DGM65QTKTLWGTRSA, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.862850518 (515h30m36.733933553s) [noise=true , address=145.79.227.39:4013 , piece_id=IGFHVLKGL55LPXB7ZXGZEMJ24NXXWIZDMYEXQVS6CW4W5Z36SG6A, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:36.885197181 (515h30m36.756280216s) [noise=true , address=145.79.227.5:4222 , piece_id=VGS73ZMHOSLNRB4VJURJ4AFCPGB3OTGHCEIBZKRUGXACJ7JBKN7A, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.098722749 (515h30m36.969805784s) [noise=true , address=45.91.116.14:4189 , piece_id=Q6CO4VRZE7QQGC7O4WUPUMFPWTGNTL6ZX6CHZZRJ4VM6DNDAXYRA, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.102372940 (515h30m36.973455985s) [noise=true , address=45.91.116.5:4298 , piece_id=AZNIH3BVMIHI6AGMLYHELREOCOF7XZXGY5BZU653QQN5OZUKJ2MA, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.108195417 (515h30m36.979278452s) [noise=true , address=45.91.116.11:4118 , piece_id=QIN3KJ4KEEZF5GOSVKNFKX6D45DQDRMLLAUBHXDIW2D7MO45LOWQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.168834957 (515h30m37.039917992s) [noise=true , address=45.91.116.4:4939 , piece_id=VP6YHPMIPHTHZRF7WO3YKKQJ4FBPD4BDAUD6YD2ZMFQIW7V2FMQQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.206383899 (515h30m37.077466944s) [noise=true , address=45.91.116.8:4048 , piece_id=EYUG6K4FJIKZLXUIBRKBF4RTAQQRZG3VHTTK6NRQKVOQGRZC5ZFA, seg_pos={0 0}, upload=452318]: Done uploading piece. err: <nil> Dec 27 02:59:37.206404237 (515h30m37.077487282s) [seg_pos={0 0}, upload=452318]: Segment reached optimal threshold of 13 pieces. Dec 27 02:59:37.206653902 (515h30m37.077736947s) [noise=true , address=145.79.227.33:4871 , piece_id=UI75MILQ4RUOGSLSGEQ3TYVBFSDS7BDSA5SC6QVNG7MTHKDS2SDQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 02:59:37.206660416 (515h30m37.077743461s) [noise=true , address=145.79.227.33:4871 , piece_id=UI75MILQ4RUOGSLSGEQ3TYVBFSDS7BDSA5SC6QVNG7MTHKDS2SDQ, seg_pos={0 0}, upload=452318]: Long tail cancellation due to optimal threshold Dec 27 02:59:37.206937634 (515h30m37.078020679s) [noise=true , address=45.91.116.7:4012 , piece_id=XN3GANEWJQ6ABXY2VP5ZKECIPPMHXXUOOOWNYN443XKGNRKRZNYQ, seg_pos={0 0}, upload=452318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: read tcp 188.166.225.170:38046->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 02:59:37.206944747 (515h30m37.078027792s) [noise=true , address=45.91.116.7:4012 , piece_id=XN3GANEWJQ6ABXY2VP5ZKECIPPMHXXUOOOWNYN443XKGNRKRZNYQ, seg_pos={0 0}, upload=452318]: Long tail cancellation due to optimal threshold Dec 27 02:59:37.206955046 (515h30m37.078038081s) [noise=true , address=145.79.227.16:4912 , piece_id=MSZJWTMKLH3NHR3D62P6ZUP5DQPCSWVFCFG3A6LKEAYLLVR34Y7Q, seg_pos={0 0}, upload=452318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:53314->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:53314->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53314->145.79.227.16:4912: use of closed network connection Dec 27 02:59:37.206970415 (515h30m37.07805345s) [noise=true , address=145.79.227.16:4912 , piece_id=MSZJWTMKLH3NHR3D62P6ZUP5DQPCSWVFCFG3A6LKEAYLLVR34Y7Q, seg_pos={0 0}, upload=452318]: Long tail cancellation due to optimal threshold Dec 27 02:59:37.206992988 (515h30m37.078076023s) [seg_pos={0 0}, upload=452318]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:59:37.207109932 (515h30m37.078192967s) [upload=452318]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:59:37.207116213 (515h30m37.078199248s) [upload=452318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:59:37.504483153 (515h30m37.375566208s) [upload=452318]: Done uploading segments... Dec 27 02:59:37.504505516 (515h30m37.375588561s) [upload=452318]: Upload finished. err: <nil> Duration: 30.006197536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 24.669352962s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:59:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 24.669379463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.X] duration: 30.002523476s trace: https://jaeger.dev.storj.io/trace/69cf3278193a26a4 FAIL (2025-12-27 02:59:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: G4XKA43JL7QPC7LT4CRIRXKMQ23PAJWOL45B6FPPD2KCOWWFLTLQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: T2FTICQZO4737X54GLQO5Y4YS2RZWTRWRHRXBC4RIO5BTQBS5DKA) context deadline exceeded ======LOGS====== ments... Dec 27 02:58:45.799230434 (515h29m45.670313479s) [upload=452306]: Got next segment. Inline: false Dec 27 02:58:45.799260221 (515h29m45.670343256s) [upload=452306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:58:45.799264589 (515h29m45.670347624s) [upload=452306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:58:46.040778922 (515h29m45.911861957s) [seg_pos={0 0}, upload=452306]: Begin upload segment... Dec 27 02:58:46.040822515 (515h29m45.91190556s) [seg_pos={0 0}, upload=452306]: Done begin upload segment. Dec 27 02:58:46.040848394 (515h29m45.911931429s) [noise=true , address=145.79.227.36:4519 , piece_id=6AOIVV7YKLKC4YWIDM3GN33LLRB44HNS4VL3XZXXZ53DACQFLZBQ, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.041011154 (515h29m45.912094199s) [noise=true , address=145.79.227.47:4267 , piece_id=7HVK56HBJVBFD6ALL4WJBZYRTDKKMG3PUBQBYT7X3H5GTTQTN63A, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.041716287 (515h29m45.912799332s) [noise=true , address=45.91.116.8:4048 , piece_id=GT4WMGCGH7KOQHUVKVGIAAWE2T5VNWYKIJDYZHPXUE3DPCYP2KJQ, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.041999256 (515h29m45.913082291s) [noise=true , address=145.79.227.8:4331 , piece_id=JYGGVA6QD5PFWF3WQUBRDBT4JKEOPK57HMNY67HWNMZMWMEMXIYA, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.042303404 (515h29m45.913386449s) [noise=true , address=45.91.116.12:4225 , piece_id=MBU4QVQJIG7X4EZN6Z7M6QF77CVNFV4TM2FDCR3L6EYLVFJPZF7Q, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.042338441 (515h29m45.913421476s) [noise=true , address=45.91.116.10:4082 , piece_id=G4XKA43JL7QPC7LT4CRIRXKMQ23PAJWOL45B6FPPD2KCOWWFLTLQ, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.042673289 (515h29m45.913756324s) [noise=true , address=77.90.31.11:4554 , piece_id=47O4DFQXCSWPYGDIPXXWVSHYUV2S5GFJPITYVSKN3NDW4OJDGS6A, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.042885202 (515h29m45.913968237s) [noise=true , address=77.90.31.15:4700 , piece_id=RA6IDIBYB5D26LLSS447ARKXKOVSZH4REW4ILX62I5I6Q5KEXN3A, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.042936440 (515h29m45.914019485s) [noise=true , address=145.79.227.14:4838 , piece_id=K7DBYGIPXMDINAMXWJNFGN7W5LBEROMEUPBEDK7U4ZFEW24LFK3Q, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.043204921 (515h29m45.914287956s) [noise=true , address=145.79.227.3:4151 , piece_id=6P7YLJTUZXPGHHVC3N5IFQGKSGVD6ZOKTSGTOX7WXTS6HV7RPVTQ, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.043431032 (515h29m45.914514077s) [noise=true , address=145.79.227.12:4765 , piece_id=QDIYHPSNSL4G5BLIRHGCIF2MDMNH5GNIQJPVPDU7K5MQFSNNLO4Q, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.043535621 (515h29m45.914618666s) [noise=true , address=145.79.227.43:4157 , piece_id=OVHMMAANKUKNK2PEBRJNEX3VHWQA27CRYOXFA56Q6IKR4X3XMC6Q, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.043830883 (515h29m45.914913918s) [noise=true , address=145.79.227.51:4411 , piece_id=K4TAFYLCM56TVLQHUBFVRE2AULXYVTUZ6VWHECV7OMJLO3VHGFFA, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.044070750 (515h29m45.915153785s) [noise=true , address=77.90.31.21:4880 , piece_id=T2FTICQZO4737X54GLQO5Y4YS2RZWTRWRHRXBC4RIO5BTQBS5DKA, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.044749883 (515h29m45.915832929s) [noise=true , address=45.91.116.14:4189 , piece_id=Z3EXZCO7TILYV3BK5FPDVDW2JMKFI2J7B26BN2MFZYMUBZ6FHF5A, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:46.045483570 (515h29m45.916566605s) [noise=true , address=45.91.116.5:4298 , piece_id=T56DZ3FITR3BLSUDT362RQL4ZI2SYND7L7E52HGCXWOIXWLUWGKA, seg_pos={0 0}, upload=452306]: Uploading piece... Dec 27 02:58:47.651860987 (515h29m47.522944022s) [upload=452306]: Next returned nil segment Dec 27 02:58:47.651879462 (515h29m47.522962497s) [upload=452306]: Waiting for error group managing segments... Dec 27 02:58:48.036014654 (515h29m47.907097699s) [noise=true , address=145.79.227.51:4411 , piece_id=K4TAFYLCM56TVLQHUBFVRE2AULXYVTUZ6VWHECV7OMJLO3VHGFFA, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.036349721 (515h29m47.907432756s) [noise=true , address=145.79.227.14:4838 , piece_id=K7DBYGIPXMDINAMXWJNFGN7W5LBEROMEUPBEDK7U4ZFEW24LFK3Q, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.044288378 (515h29m47.915371413s) [noise=true , address=145.79.227.43:4157 , piece_id=OVHMMAANKUKNK2PEBRJNEX3VHWQA27CRYOXFA56Q6IKR4X3XMC6Q, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.046986556 (515h29m47.918069611s) [noise=true , address=145.79.227.36:4519 , piece_id=6AOIVV7YKLKC4YWIDM3GN33LLRB44HNS4VL3XZXXZ53DACQFLZBQ, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.048458579 (515h29m47.919541614s) [noise=true , address=145.79.227.3:4151 , piece_id=6P7YLJTUZXPGHHVC3N5IFQGKSGVD6ZOKTSGTOX7WXTS6HV7RPVTQ, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.059283001 (515h29m47.930366036s) [noise=true , address=145.79.227.47:4267 , piece_id=7HVK56HBJVBFD6ALL4WJBZYRTDKKMG3PUBQBYT7X3H5GTTQTN63A, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.067715317 (515h29m47.938798363s) [noise=true , address=145.79.227.12:4765 , piece_id=QDIYHPSNSL4G5BLIRHGCIF2MDMNH5GNIQJPVPDU7K5MQFSNNLO4Q, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.232629668 (515h29m48.103712713s) [noise=true , address=77.90.31.11:4554 , piece_id=47O4DFQXCSWPYGDIPXXWVSHYUV2S5GFJPITYVSKN3NDW4OJDGS6A, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.251628446 (515h29m48.122711511s) [noise=true , address=77.90.31.21:4880 , piece_id=T2FTICQZO4737X54GLQO5Y4YS2RZWTRWRHRXBC4RIO5BTQBS5DKA, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.317774832 (515h29m48.188857867s) [noise=true , address=145.79.227.8:4331 , piece_id=JYGGVA6QD5PFWF3WQUBRDBT4JKEOPK57HMNY67HWNMZMWMEMXIYA, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.372310959 (515h29m48.243393994s) [noise=true , address=45.91.116.10:4082 , piece_id=G4XKA43JL7QPC7LT4CRIRXKMQ23PAJWOL45B6FPPD2KCOWWFLTLQ, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.403215768 (515h29m48.274298803s) [noise=true , address=45.91.116.14:4189 , piece_id=Z3EXZCO7TILYV3BK5FPDVDW2JMKFI2J7B26BN2MFZYMUBZ6FHF5A, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.443325345 (515h29m48.31440838s) [noise=true , address=45.91.116.5:4298 , piece_id=T56DZ3FITR3BLSUDT362RQL4ZI2SYND7L7E52HGCXWOIXWLUWGKA, seg_pos={0 0}, upload=452306]: Done uploading piece. err: <nil> Dec 27 02:58:48.443344321 (515h29m48.314427356s) [seg_pos={0 0}, upload=452306]: Segment reached optimal threshold of 13 pieces. Dec 27 02:58:48.443516940 (515h29m48.314599975s) [noise=true , address=45.91.116.8:4048 , piece_id=GT4WMGCGH7KOQHUVKVGIAAWE2T5VNWYKIJDYZHPXUE3DPCYP2KJQ, seg_pos={0 0}, upload=452306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:34388->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:34388->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34388->45.91.116.8:4048: use of closed network connection Dec 27 02:58:48.443529724 (515h29m48.314612759s) [noise=true , address=45.91.116.8:4048 , piece_id=GT4WMGCGH7KOQHUVKVGIAAWE2T5VNWYKIJDYZHPXUE3DPCYP2KJQ, seg_pos={0 0}, upload=452306]: Long tail cancellation due to optimal threshold Dec 27 02:58:48.443620397 (515h29m48.314703442s) [noise=true , address=77.90.31.15:4700 , piece_id=RA6IDIBYB5D26LLSS447ARKXKOVSZH4REW4ILX62I5I6Q5KEXN3A, seg_pos={0 0}, upload=452306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:47294->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:47294->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47294->77.90.31.15:4700: use of closed network connection Dec 27 02:58:48.443630566 (515h29m48.314713601s) [noise=true , address=77.90.31.15:4700 , piece_id=RA6IDIBYB5D26LLSS447ARKXKOVSZH4REW4ILX62I5I6Q5KEXN3A, seg_pos={0 0}, upload=452306]: Long tail cancellation due to optimal threshold Dec 27 02:58:48.443733642 (515h29m48.314816678s) [noise=true , address=45.91.116.12:4225 , piece_id=MBU4QVQJIG7X4EZN6Z7M6QF77CVNFV4TM2FDCR3L6EYLVFJPZF7Q, seg_pos={0 0}, upload=452306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 02:58:48.443741036 (515h29m48.314824071s) [noise=true , address=45.91.116.12:4225 , piece_id=MBU4QVQJIG7X4EZN6Z7M6QF77CVNFV4TM2FDCR3L6EYLVFJPZF7Q, seg_pos={0 0}, upload=452306]: Long tail cancellation due to optimal threshold Dec 27 02:58:48.443758540 (515h29m48.314841585s) [seg_pos={0 0}, upload=452306]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:58:48.443865524 (515h29m48.314948559s) [upload=452306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:58:48.443871766 (515h29m48.314954801s) [upload=452306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:58:48.728497431 (515h29m48.599580476s) [upload=452306]: Done uploading segments... Dec 27 02:58:48.728515394 (515h29m48.599598429s) [upload=452306]: Upload finished. err: <nil> Duration: 30.002543884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 17.34179738s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 17.341812358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 27.377027405s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 27.37705634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX.] duration: 7.896938469s trace: https://jaeger.dev.storj.io/trace/3600925df0db0182 FAIL (2025-12-27 02:57:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A) context deadline exceeded ======LOGS====== ... Dec 27 02:56:30.694566573 (515h27m30.565649618s) [upload=452273]: Got next segment. Inline: false Dec 27 02:56:30.694596771 (515h27m30.565679806s) [upload=452273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:56:30.694601459 (515h27m30.565684494s) [upload=452273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:56:30.940296131 (515h27m30.811379167s) [seg_pos={0 0}, upload=452273]: Begin upload segment... Dec 27 02:56:30.940348380 (515h27m30.811431426s) [seg_pos={0 0}, upload=452273]: Done begin upload segment. Dec 27 02:56:30.940409677 (515h27m30.811492712s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.940437340 (515h27m30.811520385s) [noise=true , address=45.91.116.5:4298 , piece_id=X2HYS4VSG46UYY7ADVILHCTWPXKHUBDDQ35VCSYENQUY7VF3QSRQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.940956128 (515h27m30.812039163s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941245178 (515h27m30.812328213s) [noise=true , address=145.79.227.45:4231 , piece_id=DOF62NEEQKGNNLRNK6564EGIWHHFN6AGG63CZZYGFJ52GN7I32YQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941519441 (515h27m30.812602476s) [noise=true , address=145.79.227.50:4373 , piece_id=YAIICZJHFIYL5CH3NN2MMA3GP3MGMV5WIQSLEBYSWRWIG3QSF2OA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941779426 (515h27m30.812862461s) [noise=true , address=145.79.227.46:4603 , piece_id=ADYLKKMR64GWENMJJQUTK5ACISHZZUIX3J2T25HPWS32YDHMKJEA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.942116017 (515h27m30.813199052s) [noise=true , address=145.79.227.6:4259 , piece_id=KFQDWDBGV4DYLNZLEL5XVHKWW4ODIILRRXZ6QCW2VFUEMK5PCTCA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.942397553 (515h27m30.813480588s) [noise=true , address=45.91.116.4:4940 , piece_id=D6QRDODPCNVGMVSPITGRIGUUGGO5IV3VNA4P4CLXNXQIAGNCZ34A, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943161267 (515h27m30.814244302s) [noise=true , address=77.90.31.10:4519 , piece_id=57FNE3A4BZDDVWHPIBDC742UESJJ44ZZU32K6KM5GDWZ4J3TBPUA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943554126 (515h27m30.814637161s) [noise=true , address=145.79.227.8:4331 , piece_id=2SAHJY32TDMMQCM6A3BTP5OHSZGP2CVI35C47OI2GUJTQEI7TCQQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943859426 (515h27m30.814942471s) [noise=true , address=45.91.116.8:4045 , piece_id=6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944184766 (515h27m30.815267801s) [noise=true , address=145.79.227.49:4337 , piece_id=ZO6K37JBW5CLALL7ZJG47IYH273ISFVZPGH25I53MXTEODM5BDSA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944505146 (515h27m30.815588181s) [noise=true , address=145.79.227.39:4014 , piece_id=L675AIZ6V4W33MOHP3KIYWSN6V2LKF7YTDZBP54ITG5KICREOHJA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944795048 (515h27m30.815878083s) [noise=true , address=77.90.31.11:4553 , piece_id=CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.945114777 (515h27m30.816197812s) [noise=true , address=145.79.227.51:4409 , piece_id=AR4JZPCFD5LA6LNTKI43AH5R6QR3R6WZJKWNQHKADES7MJHJDREQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.945468100 (515h27m30.816551135s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:32.644200730 (515h27m32.515283765s) [upload=452273]: Next returned nil segment Dec 27 02:56:32.644215950 (515h27m32.515298985s) [upload=452273]: Waiting for error group managing segments... Dec 27 02:56:32.930919201 (515h27m32.802002236s) [noise=true , address=145.79.227.6:4259 , piece_id=KFQDWDBGV4DYLNZLEL5XVHKWW4ODIILRRXZ6QCW2VFUEMK5PCTCA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.931213231 (515h27m32.802296276s) [noise=true , address=145.79.227.46:4603 , piece_id=ADYLKKMR64GWENMJJQUTK5ACISHZZUIX3J2T25HPWS32YDHMKJEA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.936975152 (515h27m32.808058187s) [noise=true , address=145.79.227.49:4337 , piece_id=ZO6K37JBW5CLALL7ZJG47IYH273ISFVZPGH25I53MXTEODM5BDSA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.953395069 (515h27m32.824478104s) [noise=true , address=145.79.227.39:4014 , piece_id=L675AIZ6V4W33MOHP3KIYWSN6V2LKF7YTDZBP54ITG5KICREOHJA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.958249182 (515h27m32.829332217s) [noise=true , address=145.79.227.50:4373 , piece_id=YAIICZJHFIYL5CH3NN2MMA3GP3MGMV5WIQSLEBYSWRWIG3QSF2OA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.964890128 (515h27m32.835973163s) [noise=true , address=145.79.227.45:4231 , piece_id=DOF62NEEQKGNNLRNK6564EGIWHHFN6AGG63CZZYGFJ52GN7I32YQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.982731792 (515h27m32.853814827s) [noise=true , address=145.79.227.51:4409 , piece_id=AR4JZPCFD5LA6LNTKI43AH5R6QR3R6WZJKWNQHKADES7MJHJDREQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.063155505 (515h27m32.93423854s) [noise=true , address=145.79.227.8:4331 , piece_id=2SAHJY32TDMMQCM6A3BTP5OHSZGP2CVI35C47OI2GUJTQEI7TCQQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.175042441 (515h27m33.046125476s) [noise=true , address=45.91.116.8:4045 , piece_id=6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.279804348 (515h27m33.150887383s) [noise=true , address=45.91.116.4:4940 , piece_id=D6QRDODPCNVGMVSPITGRIGUUGGO5IV3VNA4P4CLXNXQIAGNCZ34A, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.900269520 (515h27m33.771352555s) [noise=true , address=77.90.31.10:4519 , piece_id=57FNE3A4BZDDVWHPIBDC742UESJJ44ZZU32K6KM5GDWZ4J3TBPUA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.020080194 (515h27m33.891163239s) [noise=true , address=45.91.116.5:4298 , piece_id=X2HYS4VSG46UYY7ADVILHCTWPXKHUBDDQ35VCSYENQUY7VF3QSRQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.047324372 (515h27m33.918407417s) [noise=true , address=77.90.31.11:4553 , piece_id=CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.047375740 (515h27m33.918458775s) [seg_pos={0 0}, upload=452273]: Segment reached optimal threshold of 13 pieces. Dec 27 02:56:34.047627930 (515h27m33.918710985s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 27 02:56:34.047638961 (515h27m33.918722006s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047650373 (515h27m33.918733418s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection Dec 27 02:56:34.047666153 (515h27m33.918749188s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047940796 (515h27m33.919023841s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection Dec 27 02:56:34.047953139 (515h27m33.919036194s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047978377 (515h27m33.919061412s) [seg_pos={0 0}, upload=452273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:56:34.048068148 (515h27m33.919151194s) [upload=452273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:56:34.048075973 (515h27m33.919159018s) [upload=452273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:56:34.352934876 (515h27m34.224017912s) [upload=452273]: Done uploading segments... Dec 27 02:56:34.352954874 (515h27m34.224037909s) [upload=452273]: Upload finished. err: <nil> Duration: 7.896964467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 40.652603484s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:57:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 40.652628983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXXX] duration: 30.003406139s trace: https://jaeger.dev.storj.io/trace/3600925df0db0182 FAIL (2025-12-27 02:57:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A) context deadline exceeded ======LOGS====== ... Dec 27 02:56:30.694566573 (515h27m30.565649618s) [upload=452273]: Got next segment. Inline: false Dec 27 02:56:30.694596771 (515h27m30.565679806s) [upload=452273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:56:30.694601459 (515h27m30.565684494s) [upload=452273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:56:30.940296131 (515h27m30.811379167s) [seg_pos={0 0}, upload=452273]: Begin upload segment... Dec 27 02:56:30.940348380 (515h27m30.811431426s) [seg_pos={0 0}, upload=452273]: Done begin upload segment. Dec 27 02:56:30.940409677 (515h27m30.811492712s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.940437340 (515h27m30.811520385s) [noise=true , address=45.91.116.5:4298 , piece_id=X2HYS4VSG46UYY7ADVILHCTWPXKHUBDDQ35VCSYENQUY7VF3QSRQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.940956128 (515h27m30.812039163s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941245178 (515h27m30.812328213s) [noise=true , address=145.79.227.45:4231 , piece_id=DOF62NEEQKGNNLRNK6564EGIWHHFN6AGG63CZZYGFJ52GN7I32YQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941519441 (515h27m30.812602476s) [noise=true , address=145.79.227.50:4373 , piece_id=YAIICZJHFIYL5CH3NN2MMA3GP3MGMV5WIQSLEBYSWRWIG3QSF2OA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.941779426 (515h27m30.812862461s) [noise=true , address=145.79.227.46:4603 , piece_id=ADYLKKMR64GWENMJJQUTK5ACISHZZUIX3J2T25HPWS32YDHMKJEA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.942116017 (515h27m30.813199052s) [noise=true , address=145.79.227.6:4259 , piece_id=KFQDWDBGV4DYLNZLEL5XVHKWW4ODIILRRXZ6QCW2VFUEMK5PCTCA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.942397553 (515h27m30.813480588s) [noise=true , address=45.91.116.4:4940 , piece_id=D6QRDODPCNVGMVSPITGRIGUUGGO5IV3VNA4P4CLXNXQIAGNCZ34A, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943161267 (515h27m30.814244302s) [noise=true , address=77.90.31.10:4519 , piece_id=57FNE3A4BZDDVWHPIBDC742UESJJ44ZZU32K6KM5GDWZ4J3TBPUA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943554126 (515h27m30.814637161s) [noise=true , address=145.79.227.8:4331 , piece_id=2SAHJY32TDMMQCM6A3BTP5OHSZGP2CVI35C47OI2GUJTQEI7TCQQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.943859426 (515h27m30.814942471s) [noise=true , address=45.91.116.8:4045 , piece_id=6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944184766 (515h27m30.815267801s) [noise=true , address=145.79.227.49:4337 , piece_id=ZO6K37JBW5CLALL7ZJG47IYH273ISFVZPGH25I53MXTEODM5BDSA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944505146 (515h27m30.815588181s) [noise=true , address=145.79.227.39:4014 , piece_id=L675AIZ6V4W33MOHP3KIYWSN6V2LKF7YTDZBP54ITG5KICREOHJA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.944795048 (515h27m30.815878083s) [noise=true , address=77.90.31.11:4553 , piece_id=CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.945114777 (515h27m30.816197812s) [noise=true , address=145.79.227.51:4409 , piece_id=AR4JZPCFD5LA6LNTKI43AH5R6QR3R6WZJKWNQHKADES7MJHJDREQ, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:30.945468100 (515h27m30.816551135s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Uploading piece... Dec 27 02:56:32.644200730 (515h27m32.515283765s) [upload=452273]: Next returned nil segment Dec 27 02:56:32.644215950 (515h27m32.515298985s) [upload=452273]: Waiting for error group managing segments... Dec 27 02:56:32.930919201 (515h27m32.802002236s) [noise=true , address=145.79.227.6:4259 , piece_id=KFQDWDBGV4DYLNZLEL5XVHKWW4ODIILRRXZ6QCW2VFUEMK5PCTCA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.931213231 (515h27m32.802296276s) [noise=true , address=145.79.227.46:4603 , piece_id=ADYLKKMR64GWENMJJQUTK5ACISHZZUIX3J2T25HPWS32YDHMKJEA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.936975152 (515h27m32.808058187s) [noise=true , address=145.79.227.49:4337 , piece_id=ZO6K37JBW5CLALL7ZJG47IYH273ISFVZPGH25I53MXTEODM5BDSA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.953395069 (515h27m32.824478104s) [noise=true , address=145.79.227.39:4014 , piece_id=L675AIZ6V4W33MOHP3KIYWSN6V2LKF7YTDZBP54ITG5KICREOHJA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.958249182 (515h27m32.829332217s) [noise=true , address=145.79.227.50:4373 , piece_id=YAIICZJHFIYL5CH3NN2MMA3GP3MGMV5WIQSLEBYSWRWIG3QSF2OA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.964890128 (515h27m32.835973163s) [noise=true , address=145.79.227.45:4231 , piece_id=DOF62NEEQKGNNLRNK6564EGIWHHFN6AGG63CZZYGFJ52GN7I32YQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:32.982731792 (515h27m32.853814827s) [noise=true , address=145.79.227.51:4409 , piece_id=AR4JZPCFD5LA6LNTKI43AH5R6QR3R6WZJKWNQHKADES7MJHJDREQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.063155505 (515h27m32.93423854s) [noise=true , address=145.79.227.8:4331 , piece_id=2SAHJY32TDMMQCM6A3BTP5OHSZGP2CVI35C47OI2GUJTQEI7TCQQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.175042441 (515h27m33.046125476s) [noise=true , address=45.91.116.8:4045 , piece_id=6N72Q6PVJYWDSCBAXGNRG4WQF3IRCHNOYWYYUZO5YKVQNKEBZMOQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.279804348 (515h27m33.150887383s) [noise=true , address=45.91.116.4:4940 , piece_id=D6QRDODPCNVGMVSPITGRIGUUGGO5IV3VNA4P4CLXNXQIAGNCZ34A, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:33.900269520 (515h27m33.771352555s) [noise=true , address=77.90.31.10:4519 , piece_id=57FNE3A4BZDDVWHPIBDC742UESJJ44ZZU32K6KM5GDWZ4J3TBPUA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.020080194 (515h27m33.891163239s) [noise=true , address=45.91.116.5:4298 , piece_id=X2HYS4VSG46UYY7ADVILHCTWPXKHUBDDQ35VCSYENQUY7VF3QSRQ, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.047324372 (515h27m33.918407417s) [noise=true , address=77.90.31.11:4553 , piece_id=CW3RNZA4CKT43QNBORODNVCYR2QZBHYD7NOVZR6J6XZO324DZL3A, seg_pos={0 0}, upload=452273]: Done uploading piece. err: <nil> Dec 27 02:56:34.047375740 (515h27m33.918458775s) [seg_pos={0 0}, upload=452273]: Segment reached optimal threshold of 13 pieces. Dec 27 02:56:34.047627930 (515h27m33.918710985s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 27 02:56:34.047638961 (515h27m33.918722006s) [noise=true , address=145.79.227.5:4222 , piece_id=M3HHBD2GOMMGZTOBRCOPPKZMNH4B4F5PRCOBMIBY6CDM262CWYSA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047650373 (515h27m33.918733418s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49094->77.90.31.13:4627: use of closed network connection Dec 27 02:56:34.047666153 (515h27m33.918749188s) [noise=true , address=77.90.31.13:4627 , piece_id=FOV5QZPZXC4RFA3DK6T6RO2PWXOOYVKLCDQ6IWTRKVUH2ORYNZZA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047940796 (515h27m33.919023841s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49144->145.79.227.19:4077: use of closed network connection Dec 27 02:56:34.047953139 (515h27m33.919036194s) [noise=true , address=145.79.227.19:4077 , piece_id=7HN24I7OUUIHB2BALQOJPLUHWX264XUURORZYU2OZHTSDLKOO7IA, seg_pos={0 0}, upload=452273]: Long tail cancellation due to optimal threshold Dec 27 02:56:34.047978377 (515h27m33.919061412s) [seg_pos={0 0}, upload=452273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:56:34.048068148 (515h27m33.919151194s) [upload=452273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:56:34.048075973 (515h27m33.919159018s) [upload=452273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:56:34.352934876 (515h27m34.224017912s) [upload=452273]: Done uploading segments... Dec 27 02:56:34.352954874 (515h27m34.224037909s) [upload=452273]: Upload finished. err: <nil> Duration: 30.003430105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 31.651332417s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:56:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 31.651359017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXXX] duration: 30.002916818s trace: https://jaeger.dev.storj.io/trace/e9cf6570c15180b FAIL (2025-12-27 02:55:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 02:55:29.656695623 (515h26m29.527778658s) [upload=452262]: Starting upload Dec 27 02:55:29.656753662 (515h26m29.527836697s) [upload=452262]: Done starting upload Dec 27 02:55:29.656874453 (515h26m29.527957488s) [upload=452262]: Uploading segments... Dec 27 02:55:29.656967530 (515h26m29.528050566s) [upload=452262]: Got next segment. Inline: false Dec 27 02:55:29.657008719 (515h26m29.528091754s) [upload=452262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:55:29.657013888 (515h26m29.528096923s) [upload=452262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:55:29.903137818 (515h26m29.774220853s) [seg_pos={0 0}, upload=452262]: Begin upload segment... Dec 27 02:55:29.903208191 (515h26m29.774291227s) [seg_pos={0 0}, upload=452262]: Done begin upload segment. Dec 27 02:55:29.903281031 (515h26m29.774364086s) [noise=true , address=145.79.227.41:4085 , piece_id=5BZA3AQZ3TJ2K4MKSPVJGBM3R7YHBYOSGKHPGXBR56GWW62VMEHA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.903335043 (515h26m29.774418078s) [noise=true , address=45.91.116.13:4155 , piece_id=FERT3E5VX7FYT4ZIKFI3PEHRQLK5CGI54EJHBOI626HTYJTN3EWA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.904542102 (515h26m29.775625137s) [noise=true , address=145.79.227.6:4257 , piece_id=ED5JRO65BYP6TZCFVH2CNH4JSV5YDYGTG7SMZKGRUVESMEG6JQTA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.904730410 (515h26m29.775813445s) [noise=true , address=145.79.227.5:4222 , piece_id=VNQXM3HCVVQM2GGV4PZDCFNFMQWEC7FG5X2LPWEVMH7FPMYGURZQ, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905260980 (515h26m29.776344015s) [noise=true , address=145.79.227.47:4267 , piece_id=YY56QJDR3AUP5JWO2LVBA3C2CMEHGZ5W3G7CDAI5QYEVCC4DUEQA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905299925 (515h26m29.77638296s) [noise=true , address=145.79.227.12:4765 , piece_id=53PLNTH7V2ACEZLYBHETKV3C3YZPKCCBDJD7LISAMYIJSNKV2ECA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905483705 (515h26m29.77656674s) [noise=true , address=145.79.227.45:4230 , piece_id=3EROJULNFJMLOREQKVUEAGSF6RXNHTXDHC7MV77V5KWQSUVQ4Y3Q, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905532968 (515h26m29.776616004s) [noise=true , address=145.79.227.30:4761 , piece_id=EGH3VIQZG75J6XPGXYNY4WU7DPCLAJ6RTZYIUZXAE7M2Q4H7G6ZA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905565220 (515h26m29.776648265s) [noise=true , address=45.91.116.10:4084 , piece_id=SM4FNVEXU2YLACLBR3542TBYC62D7YFRZVP3FZ23O7L6DRBDMROA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905594486 (515h26m29.776677531s) [noise=true , address=145.79.227.46:4603 , piece_id=CKRAHY5PHSVZCLL3IA7AI5JHUBTUDMDIOI35AMRKJTGZAE4DK7TA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905622249 (515h26m29.776705284s) [noise=true , address=145.79.227.52:4445 , piece_id=FMJSYGENRCF62THYTODH2AFXFKXOCGF24JTMCQM2CAG6YU7YJMQQ, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905646765 (515h26m29.77672981s) [noise=true , address=45.91.116.11:4120 , piece_id=PKURPQP2XJLFVDFC3WEYPTON3AZGBDUHUL6UJLPI2NN5NM2Y2GOQ, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905671402 (515h26m29.776754437s) [noise=true , address=45.91.116.12:4227 , piece_id=NDPCD757VFBVRXBMJLOLEEDWPGBDH5LP5SSMZ2XLLKURTAIVDKTQ, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905699165 (515h26m29.77678221s) [noise=true , address=77.90.31.15:4697 , piece_id=NSCLYYOGJUAPFMAFFMJIXAWIXOIQEGVN25DIL77Q7WEUGMQ3JRMQ, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905727058 (515h26m29.776810094s) [noise=true , address=145.79.227.51:4410 , piece_id=OE5YRP45GTFZRZOQ3SL3B4XYQGCJZVVW3V6MBRDXV2VUAAXH5S3Q, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:29.905887594 (515h26m29.776970629s) [noise=true , address=145.79.227.48:4301 , piece_id=2FTJKEV6JQ7WPXXXSIHQG4MIT4AD2MUN2PHR7Z2426ANM3XBR4QA, seg_pos={0 0}, upload=452262]: Uploading piece... Dec 27 02:55:31.610692137 (515h26m31.481775172s) [upload=452262]: Next returned nil segment Dec 27 02:55:31.610709640 (515h26m31.481792675s) [upload=452262]: Waiting for error group managing segments... Dec 27 02:55:31.891522494 (515h26m31.762605539s) [noise=true , address=145.79.227.41:4085 , piece_id=5BZA3AQZ3TJ2K4MKSPVJGBM3R7YHBYOSGKHPGXBR56GWW62VMEHA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.893623726 (515h26m31.764706761s) [noise=true , address=145.79.227.5:4222 , piece_id=VNQXM3HCVVQM2GGV4PZDCFNFMQWEC7FG5X2LPWEVMH7FPMYGURZQ, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.914634044 (515h26m31.785717079s) [noise=true , address=145.79.227.46:4603 , piece_id=CKRAHY5PHSVZCLL3IA7AI5JHUBTUDMDIOI35AMRKJTGZAE4DK7TA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.917191424 (515h26m31.788274469s) [noise=true , address=145.79.227.45:4230 , piece_id=3EROJULNFJMLOREQKVUEAGSF6RXNHTXDHC7MV77V5KWQSUVQ4Y3Q, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.923322298 (515h26m31.794405343s) [noise=true , address=145.79.227.47:4267 , piece_id=YY56QJDR3AUP5JWO2LVBA3C2CMEHGZ5W3G7CDAI5QYEVCC4DUEQA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.923839192 (515h26m31.794922227s) [noise=true , address=145.79.227.6:4257 , piece_id=ED5JRO65BYP6TZCFVH2CNH4JSV5YDYGTG7SMZKGRUVESMEG6JQTA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.925659438 (515h26m31.796742493s) [noise=true , address=145.79.227.12:4765 , piece_id=53PLNTH7V2ACEZLYBHETKV3C3YZPKCCBDJD7LISAMYIJSNKV2ECA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.926053739 (515h26m31.797136794s) [noise=true , address=145.79.227.51:4410 , piece_id=OE5YRP45GTFZRZOQ3SL3B4XYQGCJZVVW3V6MBRDXV2VUAAXH5S3Q, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.932571040 (515h26m31.803654075s) [noise=true , address=145.79.227.52:4445 , piece_id=FMJSYGENRCF62THYTODH2AFXFKXOCGF24JTMCQM2CAG6YU7YJMQQ, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:31.944175115 (515h26m31.81525815s) [noise=true , address=145.79.227.48:4301 , piece_id=2FTJKEV6JQ7WPXXXSIHQG4MIT4AD2MUN2PHR7Z2426ANM3XBR4QA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:32.143414959 (515h26m32.014497994s) [noise=true , address=45.91.116.13:4155 , piece_id=FERT3E5VX7FYT4ZIKFI3PEHRQLK5CGI54EJHBOI626HTYJTN3EWA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:32.163892062 (515h26m32.034975097s) [noise=true , address=77.90.31.15:4697 , piece_id=NSCLYYOGJUAPFMAFFMJIXAWIXOIQEGVN25DIL77Q7WEUGMQ3JRMQ, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:32.194363776 (515h26m32.065446821s) [noise=true , address=45.91.116.10:4084 , piece_id=SM4FNVEXU2YLACLBR3542TBYC62D7YFRZVP3FZ23O7L6DRBDMROA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: <nil> Dec 27 02:55:32.194390156 (515h26m32.065473191s) [seg_pos={0 0}, upload=452262]: Segment reached optimal threshold of 13 pieces. Dec 27 02:55:32.194682522 (515h26m32.065765568s) [noise=true , address=45.91.116.11:4120 , piece_id=PKURPQP2XJLFVDFC3WEYPTON3AZGBDUHUL6UJLPI2NN5NM2Y2GOQ, seg_pos={0 0}, upload=452262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 02:55:32.194692241 (515h26m32.065775286s) [noise=true , address=45.91.116.11:4120 , piece_id=PKURPQP2XJLFVDFC3WEYPTON3AZGBDUHUL6UJLPI2NN5NM2Y2GOQ, seg_pos={0 0}, upload=452262]: Long tail cancellation due to optimal threshold Dec 27 02:55:32.194773877 (515h26m32.065856912s) [noise=true , address=145.79.227.30:4761 , piece_id=EGH3VIQZG75J6XPGXYNY4WU7DPCLAJ6RTZYIUZXAE7M2Q4H7G6ZA, seg_pos={0 0}, upload=452262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 02:55:32.194788624 (515h26m32.065871659s) [noise=true , address=145.79.227.30:4761 , piece_id=EGH3VIQZG75J6XPGXYNY4WU7DPCLAJ6RTZYIUZXAE7M2Q4H7G6ZA, seg_pos={0 0}, upload=452262]: Long tail cancellation due to optimal threshold Dec 27 02:55:32.194868968 (515h26m32.065952003s) [noise=true , address=45.91.116.12:4227 , piece_id=NDPCD757VFBVRXBMJLOLEEDWPGBDH5LP5SSMZ2XLLKURTAIVDKTQ, seg_pos={0 0}, upload=452262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 02:55:32.194878306 (515h26m32.065961351s) [noise=true , address=45.91.116.12:4227 , piece_id=NDPCD757VFBVRXBMJLOLEEDWPGBDH5LP5SSMZ2XLLKURTAIVDKTQ, seg_pos={0 0}, upload=452262]: Long tail cancellation due to optimal threshold Dec 27 02:55:32.194956104 (515h26m32.066039149s) [seg_pos={0 0}, upload=452262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:55:32.195046246 (515h26m32.066129281s) [upload=452262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:55:32.195053068 (515h26m32.066136113s) [upload=452262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:55:32.490111128 (515h26m32.361194173s) [upload=452262]: Done uploading segments... Dec 27 02:55:32.490129493 (515h26m32.361212528s) [upload=452262]: Upload finished. err: <nil> Duration: 30.002930643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 39.489859831s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:55:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 39.489874529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XXX] duration: 30.002374497s trace: https://jaeger.dev.storj.io/trace/64e006fd2ec41cfd FAIL (2025-12-27 02:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 02:54:53.232917643 (515h25m53.104000688s) [upload=452253]: Starting upload Dec 27 02:54:53.233071425 (515h25m53.10415448s) [upload=452253]: Done starting upload Dec 27 02:54:53.233307004 (515h25m53.104390039s) [upload=452253]: Uploading segments... Dec 27 02:54:53.233359313 (515h25m53.104442368s) [upload=452253]: Got next segment. Inline: false Dec 27 02:54:53.233397015 (515h25m53.10448006s) [upload=452253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:54:53.233403277 (515h25m53.104486323s) [upload=452253]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:54:53.475303254 (515h25m53.346386309s) [seg_pos={0 0}, upload=452253]: Begin upload segment... Dec 27 02:54:53.475361594 (515h25m53.346444639s) [seg_pos={0 0}, upload=452253]: Done begin upload segment. Dec 27 02:54:53.475374620 (515h25m53.346457655s) [noise=true , address=77.90.31.13:4629 , piece_id=QIREZAMECUWBSBJCPP7UDIKOAM44JME2UFSDXK4CI4VNFBVO5Q2Q, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.475399306 (515h25m53.346482351s) [noise=true , address=145.79.227.39:4015 , piece_id=6M4QMKGRL7A7PKKYMQWPMBRWGIEVVATTH7GHMBDLV53FELBKWKGA, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.475541597 (515h25m53.346624632s) [noise=true , address=45.91.116.14:4190 , piece_id=MQ22R3BR6E54CR4YWEMA7ACHHSH74WRLMTURK32UR5HHYU7KYA5A, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.475884119 (515h25m53.346967154s) [noise=true , address=145.79.227.46:4601 , piece_id=3LK7MCGEY76XWJ76QWD2VGICDQNTSCL2XWHJQJU25XXPPWYI5ZOQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.476188519 (515h25m53.347271564s) [noise=true , address=145.79.227.6:4259 , piece_id=YY2YPUP4TORIECGTBCEXW4WYS7KEBOJNXEI3TCW7YKGYPXYKGM3Q, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.476503659 (515h25m53.347586694s) [noise=true , address=145.79.227.11:4731 , piece_id=NO6RG7LUG5PZIU5IVBYC63CWVLNLEKOPWXQDX7UQYK7F65PVYLOA, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.476862051 (515h25m53.347945086s) [noise=true , address=145.79.227.15:4874 , piece_id=LZ6YSNK4V6CB5YS3S6DZ3ETJVBXC7XD2HI2ZH7UMQG6L7SNFSOPQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.477098772 (515h25m53.348181817s) [noise=true , address=145.79.227.45:4230 , piece_id=XEV7Z5RVGYYGYJHOE57D24BZOWG7VEDSNS4H5PYB7BLE2DQTXFSQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.477415766 (515h25m53.348498801s) [noise=true , address=145.79.227.12:4765 , piece_id=QF7I5KXP3WTCAQTD6P4EXAIIWRSYBY7RKVVMCJ6OKYQ4WIQEF34Q, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.477694196 (515h25m53.348777231s) [noise=true , address=77.90.31.21:4878 , piece_id=OXSR56QJ4DFE553CP6B5LDXO2FBYUPYFXDWHON4QO74RUYDMZMRQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.478142660 (515h25m53.349225695s) [noise=true , address=45.91.116.13:4155 , piece_id=XJTSDUYNCSLFXV7CIFQT2KALOBDBGHLJCPJ5S6HFVOTENOS6SAPQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.478664904 (515h25m53.349747939s) [noise=true , address=45.91.116.11:4118 , piece_id=IGBVDX5OOGF7POXQEMFU5MPMDH7E3YGL6FSBIWNVYFH2B45NWPZQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.480336197 (515h25m53.351419232s) [noise=true , address=45.91.116.9:4261 , piece_id=DL2F6EPFRU3RWNJF65SNMAN735LDDDT4CFMMNZ7N4LJ6VMLGLJTA, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.482384688 (515h25m53.353467733s) [noise=true , address=77.90.31.15:4697 , piece_id=RA3CARINUKUX7EJE7TMPKE7VPMHYNFSJ7Y3KN3EBEXFRJKRBITOQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.483815372 (515h25m53.354898417s) [noise=true , address=145.79.227.47:4265 , piece_id=2IXA4MJBHCFTVKXB3KNE5ZLTH6K4E7EKMHTF6IOGYMNGWCZE4CJQ, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:53.485123974 (515h25m53.356207009s) [noise=true , address=45.91.116.4:4939 , piece_id=LFOPASIZ5SHHTYT5CFARBPGIFQIY5JGJIDWEL52QW3LJ7H2234IA, seg_pos={0 0}, upload=452253]: Uploading piece... Dec 27 02:54:55.180959339 (515h25m55.052042374s) [upload=452253]: Next returned nil segment Dec 27 02:54:55.180975890 (515h25m55.052058925s) [upload=452253]: Waiting for error group managing segments... Dec 27 02:54:55.466243307 (515h25m55.337326352s) [noise=true , address=145.79.227.6:4259 , piece_id=YY2YPUP4TORIECGTBCEXW4WYS7KEBOJNXEI3TCW7YKGYPXYKGM3Q, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.471828092 (515h25m55.342911127s) [noise=true , address=145.79.227.46:4601 , piece_id=3LK7MCGEY76XWJ76QWD2VGICDQNTSCL2XWHJQJU25XXPPWYI5ZOQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.478339591 (515h25m55.349422636s) [noise=true , address=145.79.227.39:4015 , piece_id=6M4QMKGRL7A7PKKYMQWPMBRWGIEVVATTH7GHMBDLV53FELBKWKGA, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.486384370 (515h25m55.357467405s) [noise=true , address=145.79.227.45:4230 , piece_id=XEV7Z5RVGYYGYJHOE57D24BZOWG7VEDSNS4H5PYB7BLE2DQTXFSQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.491929749 (515h25m55.363012795s) [noise=true , address=145.79.227.11:4731 , piece_id=NO6RG7LUG5PZIU5IVBYC63CWVLNLEKOPWXQDX7UQYK7F65PVYLOA, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.493120467 (515h25m55.364203512s) [noise=true , address=145.79.227.15:4874 , piece_id=LZ6YSNK4V6CB5YS3S6DZ3ETJVBXC7XD2HI2ZH7UMQG6L7SNFSOPQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.501467942 (515h25m55.372550977s) [noise=true , address=145.79.227.12:4765 , piece_id=QF7I5KXP3WTCAQTD6P4EXAIIWRSYBY7RKVVMCJ6OKYQ4WIQEF34Q, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.515650939 (515h25m55.386733984s) [noise=true , address=145.79.227.47:4265 , piece_id=2IXA4MJBHCFTVKXB3KNE5ZLTH6K4E7EKMHTF6IOGYMNGWCZE4CJQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.670540661 (515h25m55.541623706s) [noise=true , address=77.90.31.21:4878 , piece_id=OXSR56QJ4DFE553CP6B5LDXO2FBYUPYFXDWHON4QO74RUYDMZMRQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.774774253 (515h25m55.645857288s) [noise=true , address=77.90.31.13:4629 , piece_id=QIREZAMECUWBSBJCPP7UDIKOAM44JME2UFSDXK4CI4VNFBVO5Q2Q, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.816126064 (515h25m55.687209099s) [noise=true , address=45.91.116.13:4155 , piece_id=XJTSDUYNCSLFXV7CIFQT2KALOBDBGHLJCPJ5S6HFVOTENOS6SAPQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.829317113 (515h25m55.700400148s) [noise=true , address=77.90.31.15:4697 , piece_id=RA3CARINUKUX7EJE7TMPKE7VPMHYNFSJ7Y3KN3EBEXFRJKRBITOQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.867455306 (515h25m55.738538341s) [noise=true , address=45.91.116.14:4190 , piece_id=MQ22R3BR6E54CR4YWEMA7ACHHSH74WRLMTURK32UR5HHYU7KYA5A, seg_pos={0 0}, upload=452253]: Done uploading piece. err: <nil> Dec 27 02:54:55.867472759 (515h25m55.738555794s) [seg_pos={0 0}, upload=452253]: Segment reached optimal threshold of 13 pieces. Dec 27 02:54:55.867656128 (515h25m55.738739173s) [noise=true , address=45.91.116.11:4118 , piece_id=IGBVDX5OOGF7POXQEMFU5MPMDH7E3YGL6FSBIWNVYFH2B45NWPZQ, seg_pos={0 0}, upload=452253]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 02:54:55.867663081 (515h25m55.738746116s) [noise=true , address=45.91.116.11:4118 , piece_id=IGBVDX5OOGF7POXQEMFU5MPMDH7E3YGL6FSBIWNVYFH2B45NWPZQ, seg_pos={0 0}, upload=452253]: Long tail cancellation due to optimal threshold Dec 27 02:54:55.867765917 (515h25m55.738848952s) [noise=true , address=45.91.116.9:4261 , piece_id=DL2F6EPFRU3RWNJF65SNMAN735LDDDT4CFMMNZ7N4LJ6VMLGLJTA, seg_pos={0 0}, upload=452253]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 02:54:55.867771198 (515h25m55.738854233s) [noise=true , address=45.91.116.9:4261 , piece_id=DL2F6EPFRU3RWNJF65SNMAN735LDDDT4CFMMNZ7N4LJ6VMLGLJTA, seg_pos={0 0}, upload=452253]: Long tail cancellation due to optimal threshold Dec 27 02:54:55.867814300 (515h25m55.738897335s) [noise=true , address=45.91.116.4:4939 , piece_id=LFOPASIZ5SHHTYT5CFARBPGIFQIY5JGJIDWEL52QW3LJ7H2234IA, seg_pos={0 0}, upload=452253]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 02:54:55.867829668 (515h25m55.738912703s) [noise=true , address=45.91.116.4:4939 , piece_id=LFOPASIZ5SHHTYT5CFARBPGIFQIY5JGJIDWEL52QW3LJ7H2234IA, seg_pos={0 0}, upload=452253]: Long tail cancellation due to optimal threshold Dec 27 02:54:55.867859175 (515h25m55.73894222s) [seg_pos={0 0}, upload=452253]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:54:55.867992288 (515h25m55.739075333s) [upload=452253]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:54:55.868004491 (515h25m55.739087566s) [upload=452253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:54:56.156579065 (515h25m56.0276621s) [upload=452253]: Done uploading segments... Dec 27 02:54:56.156599053 (515h25m56.027682088s) [upload=452253]: Upload finished. err: <nil> Duration: 30.002396809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 40.503217261s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:54:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 40.503249674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.XX] duration: 30.018236976s trace: https://jaeger.dev.storj.io/trace/64178384e2df779c FAIL (2025-12-27 02:54:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: JINW5TTEI23SRXHFG7JR6R2NP7I43WFEMGXCMEHYJRSRLE5TNTLQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 3GX3WAZKYOJHCY26WSEL4ZFVFR2KTFKODC52GTESETD773MIKK5Q) context deadline exceeded ======LOGS====== Dec 27 02:53:38.883381730 (515h24m38.754464765s) [upload=452238]: Starting upload Dec 27 02:53:38.883445702 (515h24m38.754528737s) [upload=452238]: Done starting upload Dec 27 02:53:38.883509674 (515h24m38.754592709s) [upload=452238]: Uploading segments... Dec 27 02:53:38.883633279 (515h24m38.754716324s) [upload=452238]: Got next segment. Inline: false Dec 27 02:53:38.883684877 (515h24m38.754767913s) [upload=452238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:53:38.883689967 (515h24m38.754773002s) [upload=452238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:53:39.126352375 (515h24m38.99743542s) [seg_pos={0 0}, upload=452238]: Begin upload segment... Dec 27 02:53:39.126401999 (515h24m38.997485034s) [seg_pos={0 0}, upload=452238]: Done begin upload segment. Dec 27 02:53:39.126436114 (515h24m38.997519149s) [noise=true , address=45.91.116.10:4082 , piece_id=J3KUDJ3TYROSMHXTU2LZBJP7COGLVHWHNUOUK4BTIWAHT445PJ3Q, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.126816669 (515h24m38.997899704s) [noise=true , address=45.91.116.9:4261 , piece_id=V2TS24OJOXRRC7VBRVFHIG7F3L3UTNBXS3PSYRY4GSKHNUITDW5Q, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.127319857 (515h24m38.998402892s) [noise=true , address=145.79.227.16:4911 , piece_id=EVPO6R6TR2IY3DPLFF53SQJ5BGCTLKKNKDA42SWQNOBYVVBW3SQA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.127602145 (515h24m38.99868518s) [noise=true , address=145.79.227.45:4229 , piece_id=DTKK7WIN44O3A5I5DEKIDR7VMLELBGHO7KGWGYYFSFDJMSR7PELA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.127871197 (515h24m38.998954242s) [noise=true , address=77.90.31.11:4556 , piece_id=JINW5TTEI23SRXHFG7JR6R2NP7I43WFEMGXCMEHYJRSRLE5TNTLQ, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.128170447 (515h24m38.999253482s) [noise=true , address=45.91.116.11:4120 , piece_id=3R65LQ7KV6ONB5TLT6XV3OP2EM5SLOMODGH3XS56T6FKHVCLT3JA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.128444549 (515h24m38.999527584s) [noise=true , address=45.91.116.6:4974 , piece_id=OKXBYMJRBGEZ4B5BSBJRB6Y66OKXNYKFKGY33DWB6DUVZ76IX7NA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.128696208 (515h24m38.999779243s) [noise=true , address=145.79.227.29:4943 , piece_id=APTXFGMA7UDMLHLXAAIAJO4VLRM4HRMOMZIMEL3TTQ6ONI4JYE4Q, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.129934426 (515h24m39.001017471s) [noise=true , address=145.79.227.50:4375 , piece_id=JMKIN6OVUZEFEW26ZDFQ3NKNBUSF3CD4C6MNXZDNYPWVGVBQGRQQ, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.130445939 (515h24m39.001528985s) [noise=true , address=145.79.227.47:4267 , piece_id=3RGMIBGDGLYRES7OVTJEH3R2W2BWEF3JSZLAAAO6WOT76ZQV4WBA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.130895706 (515h24m39.001978741s) [noise=true , address=145.79.227.41:4087 , piece_id=3K54CGDIUHRMLXMC4I3GVP7SRUXJACMQZJYHCO6Z5JZ6JLXXDXMQ, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.131202210 (515h24m39.002285255s) [noise=true , address=145.79.227.2:4113 , piece_id=BYWG6Y2D3PPT44DA67LHPAXA3A4KW5Y3OHO42R3LOPKYPJZCVFLQ, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.131788216 (515h24m39.002871261s) [noise=true , address=145.79.227.36:4518 , piece_id=O4R5SGOTLGWALQBQEZ7GQBZOYHOQ2DZKK6OYR5HVA3YVXGIEV6PQ, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.131895961 (515h24m39.002978996s) [noise=true , address=45.91.116.12:4227 , piece_id=3GX3WAZKYOJHCY26WSEL4ZFVFR2KTFKODC52GTESETD773MIKK5Q, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.132485744 (515h24m39.003568789s) [noise=true , address=45.91.116.7:4010 , piece_id=ZHNND4TR22CC3SSSCKP23I5NTPLH3KL6RD7TN67QJ7XX5KJWZOYA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:39.132883751 (515h24m39.003966787s) [noise=true , address=145.79.227.6:4259 , piece_id=2KOQMZ253RNQTHYUNHSSXQQEMCF6V2HJ6PNCMCNKM7U43GXJXJWA, seg_pos={0 0}, upload=452238]: Uploading piece... Dec 27 02:53:40.840520027 (515h24m40.711603072s) [upload=452238]: Next returned nil segment Dec 27 02:53:40.840532250 (515h24m40.711615285s) [upload=452238]: Waiting for error group managing segments... Dec 27 02:53:41.126874944 (515h24m40.997957989s) [noise=true , address=145.79.227.36:4518 , piece_id=O4R5SGOTLGWALQBQEZ7GQBZOYHOQ2DZKK6OYR5HVA3YVXGIEV6PQ, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.129994394 (515h24m41.001077429s) [noise=true , address=145.79.227.50:4375 , piece_id=JMKIN6OVUZEFEW26ZDFQ3NKNBUSF3CD4C6MNXZDNYPWVGVBQGRQQ, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.138992158 (515h24m41.010075193s) [noise=true , address=145.79.227.45:4229 , piece_id=DTKK7WIN44O3A5I5DEKIDR7VMLELBGHO7KGWGYYFSFDJMSR7PELA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.143617956 (515h24m41.014700992s) [noise=true , address=145.79.227.47:4267 , piece_id=3RGMIBGDGLYRES7OVTJEH3R2W2BWEF3JSZLAAAO6WOT76ZQV4WBA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.152926813 (515h24m41.024009868s) [noise=true , address=145.79.227.16:4911 , piece_id=EVPO6R6TR2IY3DPLFF53SQJ5BGCTLKKNKDA42SWQNOBYVVBW3SQA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.155932285 (515h24m41.02701532s) [noise=true , address=145.79.227.41:4087 , piece_id=3K54CGDIUHRMLXMC4I3GVP7SRUXJACMQZJYHCO6Z5JZ6JLXXDXMQ, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.156541585 (515h24m41.02762463s) [noise=true , address=145.79.227.6:4259 , piece_id=2KOQMZ253RNQTHYUNHSSXQQEMCF6V2HJ6PNCMCNKM7U43GXJXJWA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.162353864 (515h24m41.033436909s) [noise=true , address=145.79.227.2:4113 , piece_id=BYWG6Y2D3PPT44DA67LHPAXA3A4KW5Y3OHO42R3LOPKYPJZCVFLQ, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.185355914 (515h24m41.056438949s) [noise=true , address=145.79.227.29:4943 , piece_id=APTXFGMA7UDMLHLXAAIAJO4VLRM4HRMOMZIMEL3TTQ6ONI4JYE4Q, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.380515687 (515h24m41.251598722s) [noise=true , address=45.91.116.11:4120 , piece_id=3R65LQ7KV6ONB5TLT6XV3OP2EM5SLOMODGH3XS56T6FKHVCLT3JA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.384420063 (515h24m41.255503108s) [noise=true , address=77.90.31.11:4556 , piece_id=JINW5TTEI23SRXHFG7JR6R2NP7I43WFEMGXCMEHYJRSRLE5TNTLQ, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.398035389 (515h24m41.269118424s) [noise=true , address=45.91.116.7:4010 , piece_id=ZHNND4TR22CC3SSSCKP23I5NTPLH3KL6RD7TN67QJ7XX5KJWZOYA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.421921663 (515h24m41.293004708s) [noise=true , address=45.91.116.12:4227 , piece_id=3GX3WAZKYOJHCY26WSEL4ZFVFR2KTFKODC52GTESETD773MIKK5Q, seg_pos={0 0}, upload=452238]: Done uploading piece. err: <nil> Dec 27 02:53:41.421947773 (515h24m41.293030818s) [seg_pos={0 0}, upload=452238]: Segment reached optimal threshold of 13 pieces. Dec 27 02:53:41.422647385 (515h24m41.29373043s) [noise=true , address=45.91.116.9:4261 , piece_id=V2TS24OJOXRRC7VBRVFHIG7F3L3UTNBXS3PSYRY4GSKHNUITDW5Q, seg_pos={0 0}, upload=452238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:41.422701808 (515h24m41.293784853s) [noise=true , address=45.91.116.9:4261 , piece_id=V2TS24OJOXRRC7VBRVFHIG7F3L3UTNBXS3PSYRY4GSKHNUITDW5Q, seg_pos={0 0}, upload=452238]: Long tail cancellation due to optimal threshold Dec 27 02:53:41.422866232 (515h24m41.293949267s) [noise=true , address=45.91.116.6:4974 , piece_id=OKXBYMJRBGEZ4B5BSBJRB6Y66OKXNYKFKGY33DWB6DUVZ76IX7NA, seg_pos={0 0}, upload=452238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:41.422876942 (515h24m41.293959977s) [noise=true , address=45.91.116.6:4974 , piece_id=OKXBYMJRBGEZ4B5BSBJRB6Y66OKXNYKFKGY33DWB6DUVZ76IX7NA, seg_pos={0 0}, upload=452238]: Long tail cancellation due to optimal threshold Dec 27 02:53:41.422990118 (515h24m41.294073153s) [noise=true , address=45.91.116.10:4082 , piece_id=J3KUDJ3TYROSMHXTU2LZBJP7COGLVHWHNUOUK4BTIWAHT445PJ3Q, seg_pos={0 0}, upload=452238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:41.423000006 (515h24m41.294083041s) [noise=true , address=45.91.116.10:4082 , piece_id=J3KUDJ3TYROSMHXTU2LZBJP7COGLVHWHNUOUK4BTIWAHT445PJ3Q, seg_pos={0 0}, upload=452238]: Long tail cancellation due to optimal threshold Dec 27 02:53:41.423049971 (515h24m41.294133016s) [seg_pos={0 0}, upload=452238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:53:41.423222189 (515h24m41.294305234s) [upload=452238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:53:41.423230145 (515h24m41.29431318s) [upload=452238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:53:41.718611610 (515h24m41.589694645s) [upload=452238]: Done uploading segments... Dec 27 02:53:41.718629585 (515h24m41.58971262s) [upload=452238]: Upload finished. err: <nil> Duration: 30.018248029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 37.025034201s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:53:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 37.025060921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.X] duration: 30.003782846s trace: https://jaeger.dev.storj.io/trace/606f013990744ff8 FAIL (2025-12-27 02:53:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: GVXNJLBDYHIDVD762BIPSXIF4XXC6YREGUS4XVJAK2TTYEUO4DOA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: YK7YUWYQEXJIG5XKZYQCLBOPAAVBUVMH5GQFKG3LO5IBVZWCGBXA) context deadline exceeded; piecestore: (Node ID: 12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, Piece ID: UOONC3M3ZRT6DMV5YIAHIAXEBVVOQRN4IBCABIMRLJXJE5DKVOYA) context deadline exceeded ======LOGS====== Dec 27 02:53:08.640518658 (515h24m8.511601703s) [upload=452227]: Starting upload Dec 27 02:53:08.640596326 (515h24m8.511679371s) [upload=452227]: Done starting upload Dec 27 02:53:08.640680516 (515h24m8.511763561s) [upload=452227]: Uploading segments... Dec 27 02:53:08.640910294 (515h24m8.511993339s) [upload=452227]: Got next segment. Inline: false Dec 27 02:53:08.640946983 (515h24m8.512030028s) [upload=452227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:53:08.640952654 (515h24m8.512035699s) [upload=452227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:53:08.885742012 (515h24m8.756825047s) [seg_pos={0 0}, upload=452227]: Begin upload segment... Dec 27 02:53:08.885785715 (515h24m8.75686875s) [seg_pos={0 0}, upload=452227]: Done begin upload segment. Dec 27 02:53:08.885832935 (515h24m8.75691597s) [noise=true , address=145.79.227.42:4121 , piece_id=CHGDO7L7ACACF7DMWFXQGU5YXYP4ABVHCFUU6KB2NBH3EL3TQ5GA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.885889383 (515h24m8.756972418s) [noise=true , address=77.90.31.14:4663 , piece_id=NAAJUGDVTJH7V5TZEPJC7NVOGLEAJ6QO4CPKXDZKIMQXCGRU5ZCA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.886578305 (515h24m8.75766134s) [noise=true , address=77.90.31.18:4805 , piece_id=SE3TNJWSZYDMRY37LW7BFYZWWNS5X4DMJBFX7TKYD3DDPLYLDDHA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.887116960 (515h24m8.758200005s) [noise=true , address=45.91.116.5:4297 , piece_id=K4FUIP7U2XXU7M7V44Z5VMRZK3D3AYYMHN3PXYRQ3UHMCGPNCNEA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.887572738 (515h24m8.758655783s) [noise=true , address=45.91.116.8:4047 , piece_id=6XECPW2IHYLE4WPDN2PQJ44FDY4G552F2OIK2L4QEDASH5WQ4YSA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.888039527 (515h24m8.759122572s) [noise=true , address=145.79.227.46:4603 , piece_id=KKW6AOKFF4TI44YDLLZ74S5SRSG7G3AAGMCITLTZBAV3SM2F6FTQ, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.888461581 (515h24m8.759544626s) [noise=true , address=145.79.227.49:4339 , piece_id=FPWJLWBGSVMA7SNO4RDDXSQJ6SJ4ROAJ6YITLYO27WN7MU725PBQ, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.889508244 (515h24m8.760591279s) [noise=true , address=145.79.227.50:4375 , piece_id=VP6SL73ZLFI3JMMPGXQBUTQ54WKLUF4VOBYLLCMJ4R3EQT3CMSNQ, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.891268104 (515h24m8.762351149s) [noise=true , address=45.91.116.10:4084 , piece_id=GVXNJLBDYHIDVD762BIPSXIF4XXC6YREGUS4XVJAK2TTYEUO4DOA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.891306568 (515h24m8.762389603s) [noise=true , address=145.79.227.37:4565 , piece_id=P2PIRLFU3ZAHOMUPQALZPAS43BOEJS2OP5HOJ4WGNK5JTWXHA3FQ, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.894450444 (515h24m8.765533479s) [noise=true , address=145.79.227.41:4087 , piece_id=QOFCDVC3X5ZOEZHBG7CDYKGIBJHFPU23DD6EQKHW3TWAYC5Y3IYA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.900790387 (515h24m8.771873432s) [noise=true , address=45.91.116.7:4009 , piece_id=YK7YUWYQEXJIG5XKZYQCLBOPAAVBUVMH5GQFKG3LO5IBVZWCGBXA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.902155065 (515h24m8.77323811s) [noise=true , address=145.79.227.28:4727 , piece_id=UOONC3M3ZRT6DMV5YIAHIAXEBVVOQRN4IBCABIMRLJXJE5DKVOYA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.902744207 (515h24m8.773827282s) [noise=true , address=145.79.227.52:4445 , piece_id=ICY4TJULKFWERBF54LJXGXQJWCYGXHYC2YRNWCOF3VL5SW47OWGA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.906043720 (515h24m8.777126755s) [noise=true , address=145.79.227.19:4077 , piece_id=5WXLS7HF4HBYG4AVJ7XTAN3LGT5YVH2T4NASGY35JJ66SEQ35P7A, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:08.907452914 (515h24m8.778535949s) [noise=true , address=145.79.227.47:4266 , piece_id=3CZCAGZOECPIJJUJQM5UL4VMRKS5P6XKJG6ENTM7V2T6YDOXEZPA, seg_pos={0 0}, upload=452227]: Uploading piece... Dec 27 02:53:10.510174650 (515h24m10.381257685s) [upload=452227]: Next returned nil segment Dec 27 02:53:10.510192504 (515h24m10.381275549s) [upload=452227]: Waiting for error group managing segments... Dec 27 02:53:10.894674645 (515h24m10.765757691s) [noise=true , address=145.79.227.52:4445 , piece_id=ICY4TJULKFWERBF54LJXGXQJWCYGXHYC2YRNWCOF3VL5SW47OWGA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.896085983 (515h24m10.767169028s) [noise=true , address=145.79.227.46:4603 , piece_id=KKW6AOKFF4TI44YDLLZ74S5SRSG7G3AAGMCITLTZBAV3SM2F6FTQ, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.907436527 (515h24m10.778519572s) [noise=true , address=145.79.227.50:4375 , piece_id=VP6SL73ZLFI3JMMPGXQBUTQ54WKLUF4VOBYLLCMJ4R3EQT3CMSNQ, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.912677477 (515h24m10.783760522s) [noise=true , address=145.79.227.28:4727 , piece_id=UOONC3M3ZRT6DMV5YIAHIAXEBVVOQRN4IBCABIMRLJXJE5DKVOYA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.915650417 (515h24m10.786733452s) [noise=true , address=145.79.227.19:4077 , piece_id=5WXLS7HF4HBYG4AVJ7XTAN3LGT5YVH2T4NASGY35JJ66SEQ35P7A, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.923191187 (515h24m10.794274222s) [noise=true , address=145.79.227.49:4339 , piece_id=FPWJLWBGSVMA7SNO4RDDXSQJ6SJ4ROAJ6YITLYO27WN7MU725PBQ, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.923697732 (515h24m10.794780777s) [noise=true , address=145.79.227.42:4121 , piece_id=CHGDO7L7ACACF7DMWFXQGU5YXYP4ABVHCFUU6KB2NBH3EL3TQ5GA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.931127068 (515h24m10.802210113s) [noise=true , address=145.79.227.37:4565 , piece_id=P2PIRLFU3ZAHOMUPQALZPAS43BOEJS2OP5HOJ4WGNK5JTWXHA3FQ, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.932766760 (515h24m10.803849805s) [noise=true , address=145.79.227.41:4087 , piece_id=QOFCDVC3X5ZOEZHBG7CDYKGIBJHFPU23DD6EQKHW3TWAYC5Y3IYA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:10.944985788 (515h24m10.816068833s) [noise=true , address=145.79.227.47:4266 , piece_id=3CZCAGZOECPIJJUJQM5UL4VMRKS5P6XKJG6ENTM7V2T6YDOXEZPA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:11.130258164 (515h24m11.001341209s) [noise=true , address=45.91.116.10:4084 , piece_id=GVXNJLBDYHIDVD762BIPSXIF4XXC6YREGUS4XVJAK2TTYEUO4DOA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:11.138901683 (515h24m11.009984728s) [noise=true , address=45.91.116.7:4009 , piece_id=YK7YUWYQEXJIG5XKZYQCLBOPAAVBUVMH5GQFKG3LO5IBVZWCGBXA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:11.184115369 (515h24m11.055198414s) [noise=true , address=77.90.31.14:4663 , piece_id=NAAJUGDVTJH7V5TZEPJC7NVOGLEAJ6QO4CPKXDZKIMQXCGRU5ZCA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: <nil> Dec 27 02:53:11.184139776 (515h24m11.055222821s) [seg_pos={0 0}, upload=452227]: Segment reached optimal threshold of 13 pieces. Dec 27 02:53:11.184401694 (515h24m11.055484749s) [noise=true , address=45.91.116.8:4047 , piece_id=6XECPW2IHYLE4WPDN2PQJ44FDY4G552F2OIK2L4QEDASH5WQ4YSA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:11.184454234 (515h24m11.055537299s) [noise=true , address=45.91.116.8:4047 , piece_id=6XECPW2IHYLE4WPDN2PQJ44FDY4G552F2OIK2L4QEDASH5WQ4YSA, seg_pos={0 0}, upload=452227]: Long tail cancellation due to optimal threshold Dec 27 02:53:11.184571718 (515h24m11.055654753s) [noise=true , address=45.91.116.5:4297 , piece_id=K4FUIP7U2XXU7M7V44Z5VMRZK3D3AYYMHN3PXYRQ3UHMCGPNCNEA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:11.184580564 (515h24m11.055663599s) [noise=true , address=45.91.116.5:4297 , piece_id=K4FUIP7U2XXU7M7V44Z5VMRZK3D3AYYMHN3PXYRQ3UHMCGPNCNEA, seg_pos={0 0}, upload=452227]: Long tail cancellation due to optimal threshold Dec 27 02:53:11.184598449 (515h24m11.055681484s) [noise=true , address=77.90.31.18:4805 , piece_id=SE3TNJWSZYDMRY37LW7BFYZWWNS5X4DMJBFX7TKYD3DDPLYLDDHA, seg_pos={0 0}, upload=452227]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 02:53:11.184603338 (515h24m11.055686383s) [noise=true , address=77.90.31.18:4805 , piece_id=SE3TNJWSZYDMRY37LW7BFYZWWNS5X4DMJBFX7TKYD3DDPLYLDDHA, seg_pos={0 0}, upload=452227]: Long tail cancellation due to optimal threshold Dec 27 02:53:11.184622615 (515h24m11.05570565s) [seg_pos={0 0}, upload=452227]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:53:11.184704340 (515h24m11.055787385s) [upload=452227]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:53:11.184723808 (515h24m11.055806853s) [upload=452227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:53:11.472575776 (515h24m11.343658821s) [upload=452227]: Done uploading segments... Dec 27 02:53:11.472647653 (515h24m11.343730688s) [upload=452227]: Upload finished. err: <nil> Duration: 30.003809766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 42.965449225s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 42.965466196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X.] duration: 6.22352411s trace: https://jaeger.dev.storj.io/trace/45f04b75e9b35713 FAIL (2025-12-27 02:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA) context deadline exceeded ======LOGS====== c 27 02:51:25.764815931 (515h22m25.635898966s) [upload=452202]: Starting upload Dec 27 02:51:25.764871808 (515h22m25.635954843s) [upload=452202]: Done starting upload Dec 27 02:51:25.765038977 (515h22m25.636122012s) [upload=452202]: Uploading segments... Dec 27 02:51:25.765146421 (515h22m25.636229456s) [upload=452202]: Got next segment. Inline: false Dec 27 02:51:25.765182239 (515h22m25.636265284s) [upload=452202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:51:25.765187349 (515h22m25.636270394s) [upload=452202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:51:26.008970351 (515h22m25.880053396s) [seg_pos={0 0}, upload=452202]: Begin upload segment... Dec 27 02:51:26.009024072 (515h22m25.880107107s) [seg_pos={0 0}, upload=452202]: Done begin upload segment. Dec 27 02:51:26.009057817 (515h22m25.880140862s) [noise=true , address=77.90.31.15:4700 , piece_id=NBALRSXCQAIAG6D232DRYZUGDTC7PEKCERJ66BTH65HTBAIHKBFQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009077184 (515h22m25.880160239s) [noise=true , address=145.79.227.16:4912 , piece_id=4YRRZNMS2VFFVPIYVTL6DM6KW74LB4CYMHJPZBIKTNO223GOR3RQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009765635 (515h22m25.88084867s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009895902 (515h22m25.880978948s) [noise=true , address=145.79.227.50:4375 , piece_id=EUID6SHV3U4YAVGH4TGT4C426YCEZFAOCXNLFOL7L5PHBH5QF6IA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.010548736 (515h22m25.881631771s) [noise=true , address=145.79.227.5:4221 , piece_id=S2EY6XKGNWGGSKNTZY2IDVA5LRA6JQNBNTQTP2Z65HPVNTV4XAGA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011096058 (515h22m25.882179093s) [noise=true , address=45.91.116.11:4119 , piece_id=KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011338640 (515h22m25.882421675s) [noise=true , address=145.79.227.38:4978 , piece_id=SUIL6WDSKFUIUBWJEFHIJPRGID4KTGEH2XKCJXHPIT5DMTNFLFXQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011858911 (515h22m25.882941956s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012118785 (515h22m25.88320183s) [noise=true , address=145.79.227.29:4943 , piece_id=2MYQX463JEKLNVUMKTUVYH5HJNGN6YNULIURLPQKFPUH6HNOU2QQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012555177 (515h22m25.883638212s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012909762 (515h22m25.883992797s) [noise=true , address=145.79.227.46:4601 , piece_id=KMNFMH5YUNXK3VJBFWCSJPRVYB7EZO7SXOJ3IL5KRAKOSXA4LL3A, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.013330232 (515h22m25.884413267s) [noise=true , address=77.90.31.14:4663 , piece_id=AP53QUVXNP6LGUC5EY4OCTNYYDZD62ASB4ST7ZQFFA6GUY5MC6CA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.013600577 (515h22m25.884683622s) [noise=true , address=145.79.227.39:4014 , piece_id=R2QZLWLZ3LNK6FJ6E4BSF5GAKCGKIILJ2P7IWI2EGYBE5JSU5WAA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014345245 (515h22m25.88542829s) [noise=true , address=145.79.227.7:4293 , piece_id=BTFSDN3QJNMX6UMCMUV6UU7BFTI4JI23X76IMLYHEWUJYATW66JA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014709449 (515h22m25.885792494s) [noise=true , address=45.91.116.13:4154 , piece_id=JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014727212 (515h22m25.885810267s) [noise=true , address=45.91.116.12:4226 , piece_id=TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:27.632112733 (515h22m27.503195898s) [upload=452202]: Next returned nil segment Dec 27 02:51:27.632138101 (515h22m27.503221146s) [upload=452202]: Waiting for error group managing segments... Dec 27 02:51:28.017621309 (515h22m27.888704344s) [noise=true , address=145.79.227.50:4375 , piece_id=EUID6SHV3U4YAVGH4TGT4C426YCEZFAOCXNLFOL7L5PHBH5QF6IA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.017666806 (515h22m27.888749851s) [noise=true , address=145.79.227.7:4293 , piece_id=BTFSDN3QJNMX6UMCMUV6UU7BFTI4JI23X76IMLYHEWUJYATW66JA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.023266299 (515h22m27.894349344s) [noise=true , address=145.79.227.5:4221 , piece_id=S2EY6XKGNWGGSKNTZY2IDVA5LRA6JQNBNTQTP2Z65HPVNTV4XAGA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.027992548 (515h22m27.899075593s) [noise=true , address=145.79.227.38:4978 , piece_id=SUIL6WDSKFUIUBWJEFHIJPRGID4KTGEH2XKCJXHPIT5DMTNFLFXQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.030424389 (515h22m27.901507675s) [noise=true , address=145.79.227.39:4014 , piece_id=R2QZLWLZ3LNK6FJ6E4BSF5GAKCGKIILJ2P7IWI2EGYBE5JSU5WAA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.032278740 (515h22m27.903361775s) [noise=true , address=145.79.227.16:4912 , piece_id=4YRRZNMS2VFFVPIYVTL6DM6KW74LB4CYMHJPZBIKTNO223GOR3RQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.044633876 (515h22m27.915716911s) [noise=true , address=145.79.227.46:4601 , piece_id=KMNFMH5YUNXK3VJBFWCSJPRVYB7EZO7SXOJ3IL5KRAKOSXA4LL3A, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.083429762 (515h22m27.954512797s) [noise=true , address=145.79.227.29:4943 , piece_id=2MYQX463JEKLNVUMKTUVYH5HJNGN6YNULIURLPQKFPUH6HNOU2QQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.244538836 (515h22m28.115621881s) [noise=true , address=45.91.116.13:4154 , piece_id=JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.288853069 (515h22m28.159936104s) [noise=true , address=45.91.116.12:4226 , piece_id=TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.377061807 (515h22m28.248144852s) [noise=true , address=77.90.31.15:4700 , piece_id=NBALRSXCQAIAG6D232DRYZUGDTC7PEKCERJ66BTH65HTBAIHKBFQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.546296675 (515h22m28.41737971s) [noise=true , address=45.91.116.11:4119 , piece_id=KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.961336924 (515h22m28.832419959s) [noise=true , address=77.90.31.14:4663 , piece_id=AP53QUVXNP6LGUC5EY4OCTNYYDZD62ASB4ST7ZQFFA6GUY5MC6CA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.961362363 (515h22m28.832445398s) [seg_pos={0 0}, upload=452202]: Segment reached optimal threshold of 13 pieces. Dec 27 02:51:28.961577983 (515h22m28.832661038s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 02:51:28.961586880 (515h22m28.832669925s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961750021 (515h22m28.832833076s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 02:51:28.961758567 (515h22m28.832841622s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961839572 (515h22m28.832922607s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection Dec 27 02:51:28.961859639 (515h22m28.832942695s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961915677 (515h22m28.832998722s) [seg_pos={0 0}, upload=452202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:51:28.962141446 (515h22m28.833224491s) [upload=452202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:51:28.962150093 (515h22m28.833233138s) [upload=452202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:51:29.257455855 (515h22m29.12853889s) [upload=452202]: Done uploading segments... Dec 27 02:51:29.257516821 (515h22m29.128599857s) [upload=452202]: Upload finished. err: <nil> Duration: 6.22356073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 33.693226516s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 33.69326038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...X] duration: 30.002633128s trace: https://jaeger.dev.storj.io/trace/45f04b75e9b35713 FAIL (2025-12-27 02:51:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA) context deadline exceeded ======LOGS====== c 27 02:51:25.764815931 (515h22m25.635898966s) [upload=452202]: Starting upload Dec 27 02:51:25.764871808 (515h22m25.635954843s) [upload=452202]: Done starting upload Dec 27 02:51:25.765038977 (515h22m25.636122012s) [upload=452202]: Uploading segments... Dec 27 02:51:25.765146421 (515h22m25.636229456s) [upload=452202]: Got next segment. Inline: false Dec 27 02:51:25.765182239 (515h22m25.636265284s) [upload=452202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:51:25.765187349 (515h22m25.636270394s) [upload=452202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:51:26.008970351 (515h22m25.880053396s) [seg_pos={0 0}, upload=452202]: Begin upload segment... Dec 27 02:51:26.009024072 (515h22m25.880107107s) [seg_pos={0 0}, upload=452202]: Done begin upload segment. Dec 27 02:51:26.009057817 (515h22m25.880140862s) [noise=true , address=77.90.31.15:4700 , piece_id=NBALRSXCQAIAG6D232DRYZUGDTC7PEKCERJ66BTH65HTBAIHKBFQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009077184 (515h22m25.880160239s) [noise=true , address=145.79.227.16:4912 , piece_id=4YRRZNMS2VFFVPIYVTL6DM6KW74LB4CYMHJPZBIKTNO223GOR3RQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009765635 (515h22m25.88084867s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.009895902 (515h22m25.880978948s) [noise=true , address=145.79.227.50:4375 , piece_id=EUID6SHV3U4YAVGH4TGT4C426YCEZFAOCXNLFOL7L5PHBH5QF6IA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.010548736 (515h22m25.881631771s) [noise=true , address=145.79.227.5:4221 , piece_id=S2EY6XKGNWGGSKNTZY2IDVA5LRA6JQNBNTQTP2Z65HPVNTV4XAGA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011096058 (515h22m25.882179093s) [noise=true , address=45.91.116.11:4119 , piece_id=KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011338640 (515h22m25.882421675s) [noise=true , address=145.79.227.38:4978 , piece_id=SUIL6WDSKFUIUBWJEFHIJPRGID4KTGEH2XKCJXHPIT5DMTNFLFXQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.011858911 (515h22m25.882941956s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012118785 (515h22m25.88320183s) [noise=true , address=145.79.227.29:4943 , piece_id=2MYQX463JEKLNVUMKTUVYH5HJNGN6YNULIURLPQKFPUH6HNOU2QQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012555177 (515h22m25.883638212s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.012909762 (515h22m25.883992797s) [noise=true , address=145.79.227.46:4601 , piece_id=KMNFMH5YUNXK3VJBFWCSJPRVYB7EZO7SXOJ3IL5KRAKOSXA4LL3A, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.013330232 (515h22m25.884413267s) [noise=true , address=77.90.31.14:4663 , piece_id=AP53QUVXNP6LGUC5EY4OCTNYYDZD62ASB4ST7ZQFFA6GUY5MC6CA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.013600577 (515h22m25.884683622s) [noise=true , address=145.79.227.39:4014 , piece_id=R2QZLWLZ3LNK6FJ6E4BSF5GAKCGKIILJ2P7IWI2EGYBE5JSU5WAA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014345245 (515h22m25.88542829s) [noise=true , address=145.79.227.7:4293 , piece_id=BTFSDN3QJNMX6UMCMUV6UU7BFTI4JI23X76IMLYHEWUJYATW66JA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014709449 (515h22m25.885792494s) [noise=true , address=45.91.116.13:4154 , piece_id=JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:26.014727212 (515h22m25.885810267s) [noise=true , address=45.91.116.12:4226 , piece_id=TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A, seg_pos={0 0}, upload=452202]: Uploading piece... Dec 27 02:51:27.632112733 (515h22m27.503195898s) [upload=452202]: Next returned nil segment Dec 27 02:51:27.632138101 (515h22m27.503221146s) [upload=452202]: Waiting for error group managing segments... Dec 27 02:51:28.017621309 (515h22m27.888704344s) [noise=true , address=145.79.227.50:4375 , piece_id=EUID6SHV3U4YAVGH4TGT4C426YCEZFAOCXNLFOL7L5PHBH5QF6IA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.017666806 (515h22m27.888749851s) [noise=true , address=145.79.227.7:4293 , piece_id=BTFSDN3QJNMX6UMCMUV6UU7BFTI4JI23X76IMLYHEWUJYATW66JA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.023266299 (515h22m27.894349344s) [noise=true , address=145.79.227.5:4221 , piece_id=S2EY6XKGNWGGSKNTZY2IDVA5LRA6JQNBNTQTP2Z65HPVNTV4XAGA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.027992548 (515h22m27.899075593s) [noise=true , address=145.79.227.38:4978 , piece_id=SUIL6WDSKFUIUBWJEFHIJPRGID4KTGEH2XKCJXHPIT5DMTNFLFXQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.030424389 (515h22m27.901507675s) [noise=true , address=145.79.227.39:4014 , piece_id=R2QZLWLZ3LNK6FJ6E4BSF5GAKCGKIILJ2P7IWI2EGYBE5JSU5WAA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.032278740 (515h22m27.903361775s) [noise=true , address=145.79.227.16:4912 , piece_id=4YRRZNMS2VFFVPIYVTL6DM6KW74LB4CYMHJPZBIKTNO223GOR3RQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.044633876 (515h22m27.915716911s) [noise=true , address=145.79.227.46:4601 , piece_id=KMNFMH5YUNXK3VJBFWCSJPRVYB7EZO7SXOJ3IL5KRAKOSXA4LL3A, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.083429762 (515h22m27.954512797s) [noise=true , address=145.79.227.29:4943 , piece_id=2MYQX463JEKLNVUMKTUVYH5HJNGN6YNULIURLPQKFPUH6HNOU2QQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.244538836 (515h22m28.115621881s) [noise=true , address=45.91.116.13:4154 , piece_id=JF7QBZDL3GJW3SJKEKLK22G2LPN2MFAHFOJTEDMAQ6NFLHD42OBA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.288853069 (515h22m28.159936104s) [noise=true , address=45.91.116.12:4226 , piece_id=TXOE2QKFM52VYCRCH2WS5F6FLIT435QBFI7XKTZMI5EJDULERM5A, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.377061807 (515h22m28.248144852s) [noise=true , address=77.90.31.15:4700 , piece_id=NBALRSXCQAIAG6D232DRYZUGDTC7PEKCERJ66BTH65HTBAIHKBFQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.546296675 (515h22m28.41737971s) [noise=true , address=45.91.116.11:4119 , piece_id=KKQ64HQ755DJY7Q4IJ4MKIB7KZXI3JB35HNHTDC74QRO3WJZLCQA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.961336924 (515h22m28.832419959s) [noise=true , address=77.90.31.14:4663 , piece_id=AP53QUVXNP6LGUC5EY4OCTNYYDZD62ASB4ST7ZQFFA6GUY5MC6CA, seg_pos={0 0}, upload=452202]: Done uploading piece. err: <nil> Dec 27 02:51:28.961362363 (515h22m28.832445398s) [seg_pos={0 0}, upload=452202]: Segment reached optimal threshold of 13 pieces. Dec 27 02:51:28.961577983 (515h22m28.832661038s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 02:51:28.961586880 (515h22m28.832669925s) [noise=true , address=45.91.116.10:4081 , piece_id=OLBC6ZLSD2JXPJVGYR7A7HGTTZTJHIFYSNIDSSO66JUN5SBOZKJQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961750021 (515h22m28.832833076s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 02:51:28.961758567 (515h22m28.832841622s) [noise=true , address=145.79.227.44:4193 , piece_id=77UMTROAD5HEYJHNHG2XF6EBW3C7FAEKN3LLD7FDVXOGQB6J2TSQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961839572 (515h22m28.832922607s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52726->77.90.31.18:4808: use of closed network connection Dec 27 02:51:28.961859639 (515h22m28.832942695s) [noise=true , address=77.90.31.18:4808 , piece_id=IOQJTUOGOBI53URO3PR6WEAGKL2H4GM2PFMSNPRIBYTYIYHNTBQQ, seg_pos={0 0}, upload=452202]: Long tail cancellation due to optimal threshold Dec 27 02:51:28.961915677 (515h22m28.832998722s) [seg_pos={0 0}, upload=452202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:51:28.962141446 (515h22m28.833224491s) [upload=452202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:51:28.962150093 (515h22m28.833233138s) [upload=452202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:51:29.257455855 (515h22m29.12853889s) [upload=452202]: Done uploading segments... Dec 27 02:51:29.257516821 (515h22m29.128599857s) [upload=452202]: Upload finished. err: <nil> Duration: 30.00266023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.551790152s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:51:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 16.551808425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 45.157245877s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 45.157266458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX...] duration: 5.966445731s trace: https://jaeger.dev.storj.io/trace/4da53f27d3c7ab37 FAIL (2025-12-27 02:50:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA) context deadline exceeded ======LOGS====== oad=452143]: Done starting upload Dec 27 02:47:21.685990899 (515h18m21.557073944s) [upload=452143]: Uploading segments... Dec 27 02:47:21.686200819 (515h18m21.557283864s) [upload=452143]: Got next segment. Inline: false Dec 27 02:47:21.686295349 (515h18m21.557378394s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:47:21.686307632 (515h18m21.557390707s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:47:21.931055852 (515h18m21.802138907s) [seg_pos={0 0}, upload=452143]: Begin upload segment... Dec 27 02:47:21.931117430 (515h18m21.802200475s) [seg_pos={0 0}, upload=452143]: Done begin upload segment. Dec 27 02:47:21.931132097 (515h18m21.802215132s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931208302 (515h18m21.802291347s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931787064 (515h18m21.80287011s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932208026 (515h18m21.803291071s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932522725 (515h18m21.80360577s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932634838 (515h18m21.803717883s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932929410 (515h18m21.804012445s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933015273 (515h18m21.804098318s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933328549 (515h18m21.804411584s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933767055 (515h18m21.8048501s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.934193907 (515h18m21.805276952s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935177100 (515h18m21.806260135s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935728790 (515h18m21.806811835s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936249271 (515h18m21.807332316s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936492685 (515h18m21.80757573s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.937485656 (515h18m21.808568701s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:23.641517306 (515h18m23.512600351s) [upload=452143]: Next returned nil segment Dec 27 02:47:23.641537995 (515h18m23.51262104s) [upload=452143]: Waiting for error group managing segments... Dec 27 02:47:23.930682274 (515h18m23.801765309s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935029652 (515h18m23.806112687s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935285469 (515h18m23.806368514s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.936019958 (515h18m23.807103003s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.945695723 (515h18m23.816778758s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.954292863 (515h18m23.825375899s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.956009352 (515h18m23.827092397s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.964184940 (515h18m23.835267985s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.109519687 (515h18m23.980602732s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.135607433 (515h18m24.006690478s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.179009700 (515h18m24.050092745s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.191998363 (515h18m24.063081408s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.232988737 (515h18m24.104071772s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.233011751 (515h18m24.104094786s) [seg_pos={0 0}, upload=452143]: Segment reached optimal threshold of 13 pieces. Dec 27 02:47:24.233259112 (515h18m24.104342157s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection Dec 27 02:47:24.233270223 (515h18m24.104353268s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233373500 (515h18m24.104456535s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 02:47:24.233405842 (515h18m24.104488887s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233686105 (515h18m24.10476914s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection Dec 27 02:47:24.233693880 (515h18m24.104776915s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233877189 (515h18m24.104960234s) [seg_pos={0 0}, upload=452143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:47:24.234032806 (515h18m24.105115851s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:47:24.234040220 (515h18m24.105123255s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:47:24.526442354 (515h18m24.397525399s) [upload=452143]: Done uploading segments... Dec 27 02:47:24.526556932 (515h18m24.397639967s) [upload=452143]: Upload finished. err: <nil> Duration: 5.96648209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 45.724503663s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:50:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 45.724533841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX..] duration: 29.503364025s trace: https://jaeger.dev.storj.io/trace/4da53f27d3c7ab37 FAIL (2025-12-27 02:49:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA) context deadline exceeded ======LOGS====== oad=452143]: Done starting upload Dec 27 02:47:21.685990899 (515h18m21.557073944s) [upload=452143]: Uploading segments... Dec 27 02:47:21.686200819 (515h18m21.557283864s) [upload=452143]: Got next segment. Inline: false Dec 27 02:47:21.686295349 (515h18m21.557378394s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:47:21.686307632 (515h18m21.557390707s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:47:21.931055852 (515h18m21.802138907s) [seg_pos={0 0}, upload=452143]: Begin upload segment... Dec 27 02:47:21.931117430 (515h18m21.802200475s) [seg_pos={0 0}, upload=452143]: Done begin upload segment. Dec 27 02:47:21.931132097 (515h18m21.802215132s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931208302 (515h18m21.802291347s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931787064 (515h18m21.80287011s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932208026 (515h18m21.803291071s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932522725 (515h18m21.80360577s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932634838 (515h18m21.803717883s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932929410 (515h18m21.804012445s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933015273 (515h18m21.804098318s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933328549 (515h18m21.804411584s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933767055 (515h18m21.8048501s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.934193907 (515h18m21.805276952s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935177100 (515h18m21.806260135s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935728790 (515h18m21.806811835s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936249271 (515h18m21.807332316s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936492685 (515h18m21.80757573s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.937485656 (515h18m21.808568701s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:23.641517306 (515h18m23.512600351s) [upload=452143]: Next returned nil segment Dec 27 02:47:23.641537995 (515h18m23.51262104s) [upload=452143]: Waiting for error group managing segments... Dec 27 02:47:23.930682274 (515h18m23.801765309s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935029652 (515h18m23.806112687s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935285469 (515h18m23.806368514s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.936019958 (515h18m23.807103003s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.945695723 (515h18m23.816778758s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.954292863 (515h18m23.825375899s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.956009352 (515h18m23.827092397s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.964184940 (515h18m23.835267985s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.109519687 (515h18m23.980602732s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.135607433 (515h18m24.006690478s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.179009700 (515h18m24.050092745s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.191998363 (515h18m24.063081408s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.232988737 (515h18m24.104071772s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.233011751 (515h18m24.104094786s) [seg_pos={0 0}, upload=452143]: Segment reached optimal threshold of 13 pieces. Dec 27 02:47:24.233259112 (515h18m24.104342157s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection Dec 27 02:47:24.233270223 (515h18m24.104353268s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233373500 (515h18m24.104456535s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 02:47:24.233405842 (515h18m24.104488887s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233686105 (515h18m24.10476914s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection Dec 27 02:47:24.233693880 (515h18m24.104776915s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233877189 (515h18m24.104960234s) [seg_pos={0 0}, upload=452143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:47:24.234032806 (515h18m24.105115851s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:47:24.234040220 (515h18m24.105123255s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:47:24.526442354 (515h18m24.397525399s) [upload=452143]: Done uploading segments... Dec 27 02:47:24.526556932 (515h18m24.397639967s) [upload=452143]: Upload finished. err: <nil> Duration: 29.50339218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.178048206s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:49:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 15.178073153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 28.484064951s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:48:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 28.484079429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 34.997612917s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 34.99763967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX.] duration: 6.122802537s trace: https://jaeger.dev.storj.io/trace/4da53f27d3c7ab37 FAIL (2025-12-27 02:47:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA) context deadline exceeded ======LOGS====== oad=452143]: Done starting upload Dec 27 02:47:21.685990899 (515h18m21.557073944s) [upload=452143]: Uploading segments... Dec 27 02:47:21.686200819 (515h18m21.557283864s) [upload=452143]: Got next segment. Inline: false Dec 27 02:47:21.686295349 (515h18m21.557378394s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:47:21.686307632 (515h18m21.557390707s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:47:21.931055852 (515h18m21.802138907s) [seg_pos={0 0}, upload=452143]: Begin upload segment... Dec 27 02:47:21.931117430 (515h18m21.802200475s) [seg_pos={0 0}, upload=452143]: Done begin upload segment. Dec 27 02:47:21.931132097 (515h18m21.802215132s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931208302 (515h18m21.802291347s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931787064 (515h18m21.80287011s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932208026 (515h18m21.803291071s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932522725 (515h18m21.80360577s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932634838 (515h18m21.803717883s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932929410 (515h18m21.804012445s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933015273 (515h18m21.804098318s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933328549 (515h18m21.804411584s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933767055 (515h18m21.8048501s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.934193907 (515h18m21.805276952s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935177100 (515h18m21.806260135s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935728790 (515h18m21.806811835s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936249271 (515h18m21.807332316s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936492685 (515h18m21.80757573s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.937485656 (515h18m21.808568701s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:23.641517306 (515h18m23.512600351s) [upload=452143]: Next returned nil segment Dec 27 02:47:23.641537995 (515h18m23.51262104s) [upload=452143]: Waiting for error group managing segments... Dec 27 02:47:23.930682274 (515h18m23.801765309s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935029652 (515h18m23.806112687s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935285469 (515h18m23.806368514s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.936019958 (515h18m23.807103003s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.945695723 (515h18m23.816778758s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.954292863 (515h18m23.825375899s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.956009352 (515h18m23.827092397s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.964184940 (515h18m23.835267985s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.109519687 (515h18m23.980602732s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.135607433 (515h18m24.006690478s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.179009700 (515h18m24.050092745s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.191998363 (515h18m24.063081408s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.232988737 (515h18m24.104071772s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.233011751 (515h18m24.104094786s) [seg_pos={0 0}, upload=452143]: Segment reached optimal threshold of 13 pieces. Dec 27 02:47:24.233259112 (515h18m24.104342157s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection Dec 27 02:47:24.233270223 (515h18m24.104353268s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233373500 (515h18m24.104456535s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 02:47:24.233405842 (515h18m24.104488887s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233686105 (515h18m24.10476914s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection Dec 27 02:47:24.233693880 (515h18m24.104776915s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233877189 (515h18m24.104960234s) [seg_pos={0 0}, upload=452143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:47:24.234032806 (515h18m24.105115851s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:47:24.234040220 (515h18m24.105123255s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:47:24.526442354 (515h18m24.397525399s) [upload=452143]: Done uploading segments... Dec 27 02:47:24.526556932 (515h18m24.397639967s) [upload=452143]: Upload finished. err: <nil> Duration: 6.122821483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXXX] duration: 30.002251956s trace: https://jaeger.dev.storj.io/trace/4da53f27d3c7ab37 FAIL (2025-12-27 02:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA) context deadline exceeded ======LOGS====== oad=452143]: Done starting upload Dec 27 02:47:21.685990899 (515h18m21.557073944s) [upload=452143]: Uploading segments... Dec 27 02:47:21.686200819 (515h18m21.557283864s) [upload=452143]: Got next segment. Inline: false Dec 27 02:47:21.686295349 (515h18m21.557378394s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:47:21.686307632 (515h18m21.557390707s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:47:21.931055852 (515h18m21.802138907s) [seg_pos={0 0}, upload=452143]: Begin upload segment... Dec 27 02:47:21.931117430 (515h18m21.802200475s) [seg_pos={0 0}, upload=452143]: Done begin upload segment. Dec 27 02:47:21.931132097 (515h18m21.802215132s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931208302 (515h18m21.802291347s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.931787064 (515h18m21.80287011s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932208026 (515h18m21.803291071s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932522725 (515h18m21.80360577s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932634838 (515h18m21.803717883s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.932929410 (515h18m21.804012445s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933015273 (515h18m21.804098318s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933328549 (515h18m21.804411584s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.933767055 (515h18m21.8048501s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.934193907 (515h18m21.805276952s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935177100 (515h18m21.806260135s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.935728790 (515h18m21.806811835s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936249271 (515h18m21.807332316s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.936492685 (515h18m21.80757573s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:21.937485656 (515h18m21.808568701s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Uploading piece... Dec 27 02:47:23.641517306 (515h18m23.512600351s) [upload=452143]: Next returned nil segment Dec 27 02:47:23.641537995 (515h18m23.51262104s) [upload=452143]: Waiting for error group managing segments... Dec 27 02:47:23.930682274 (515h18m23.801765309s) [noise=true , address=145.79.227.3:4151 , piece_id=QXFWMFFATADOW2B6CMEWU4ROKCAGVVVLVICESNIKJAZIM6NA4VOA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935029652 (515h18m23.806112687s) [noise=true , address=145.79.227.11:4732 , piece_id=AWMJJTRKVBAFPDWVMB5YJCE3OAD255YO7X32X5KZ3Z4VPZ3DKNJA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.935285469 (515h18m23.806368514s) [noise=true , address=145.79.227.50:4373 , piece_id=VBR2PEOQ52ZYPOTYHKABJBICLNDP44H6PDMBUBGOCOOAQ4AIVWDQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.936019958 (515h18m23.807103003s) [noise=true , address=145.79.227.51:4410 , piece_id=F7W56AIZO2JU42IGZOSNDE57UC5AV4ZJ5RIXZ436QKXP4ZTKOMYQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.945695723 (515h18m23.816778758s) [noise=true , address=145.79.227.13:4804 , piece_id=O3VYY6AMOQMMQDWCLD3B2ZVIOFWPZMD6QHY2D56WTVTH5NZG4STA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.954292863 (515h18m23.825375899s) [noise=true , address=145.79.227.16:4912 , piece_id=M2DAWQPGMFS2OCWAIOLKQ5OAT7S6SZGM72ZAY5FQAM55BEX6HI5Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.956009352 (515h18m23.827092397s) [noise=true , address=145.79.227.41:4087 , piece_id=QYJDXQIPXNMWBRCVVDO6WDQL54QG3RUD66U7Z4U7NJ7X7I7XLJWA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:23.964184940 (515h18m23.835267985s) [noise=true , address=145.79.227.47:4267 , piece_id=VHC7IBAMQF2OJPSIFA23C4PO3OSUDLFGIJWTUSD27UVNUI7TPD3Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.109519687 (515h18m23.980602732s) [noise=true , address=77.90.31.12:4590 , piece_id=WPJ55YFNGGLUZXIHLFKRSLJ2SKHA24KKPVIGN7G3SGU7G7RGORYA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.135607433 (515h18m24.006690478s) [noise=true , address=77.90.31.18:4808 , piece_id=OCWLDAI2LT6I7CEV3Y4VRXRQ7R7MUIU6VAT6RVFZQBBYRNB25JSA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.179009700 (515h18m24.050092745s) [noise=true , address=45.91.116.7:4012 , piece_id=EAOOAX27ZHJJ5AM6FWH7EFFLSTGDZU7DEV3WN4J7RBWJCLDWR62A, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.191998363 (515h18m24.063081408s) [noise=true , address=45.91.116.6:4975 , piece_id=HFT2IEKHELFNJGABDM7J2ZAFAOFJOZ45SFUMTZ5ENYPBNCCMQEKA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.232988737 (515h18m24.104071772s) [noise=true , address=77.90.31.13:4629 , piece_id=5RQTX3V72HYWMCRTLJ6JQUOB4ZIZ3ND67ZKVVKJQ2WRWC6JL46IA, seg_pos={0 0}, upload=452143]: Done uploading piece. err: <nil> Dec 27 02:47:24.233011751 (515h18m24.104094786s) [seg_pos={0 0}, upload=452143]: Segment reached optimal threshold of 13 pieces. Dec 27 02:47:24.233259112 (515h18m24.104342157s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53590->145.79.227.2:4115: use of closed network connection Dec 27 02:47:24.233270223 (515h18m24.104353268s) [noise=true , address=145.79.227.2:4115 , piece_id=DJCHPVIIWIPXO6VLE2NM7ZY3SCBBQAWC4M44MJEAL575CF3H7WCQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233373500 (515h18m24.104456535s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 02:47:24.233405842 (515h18m24.104488887s) [noise=true , address=45.91.116.8:4046 , piece_id=W5E3J6LDQ76EOXXJEA4MGIU5A5PTNXXAPL2ZKEB57FM3FJDQT76Q, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233686105 (515h18m24.10476914s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:39926->145.79.227.19:4077: use of closed network connection Dec 27 02:47:24.233693880 (515h18m24.104776915s) [noise=true , address=145.79.227.19:4077 , piece_id=YC6FJUKTNXHDD6FOBBIG7SMMAF745RDCKV4S43C5XGZDBSAMKEUQ, seg_pos={0 0}, upload=452143]: Long tail cancellation due to optimal threshold Dec 27 02:47:24.233877189 (515h18m24.104960234s) [seg_pos={0 0}, upload=452143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:47:24.234032806 (515h18m24.105115851s) [upload=452143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:47:24.234040220 (515h18m24.105123255s) [upload=452143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:47:24.526442354 (515h18m24.397525399s) [upload=452143]: Done uploading segments... Dec 27 02:47:24.526556932 (515h18m24.397639967s) [upload=452143]: Upload finished. err: <nil> Duration: 30.002276341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 34.439917858s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 34.439948065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XXX] duration: 30.004451964s trace: https://jaeger.dev.storj.io/trace/5f3ced37543c193 FAIL (2025-12-27 02:47:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: XNYN4IBUUOTYWTCHYSXMVRLWCAVED4QGIWR34FST3FFTVRLIE77Q) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 4QITOQWDSNVS4KFYBOB4HKPUVFB56WKC3MLVSOV5ATA3HIJIL4RA) context deadline exceeded ======LOGS====== Dec 27 02:46:32.073872625 (515h17m31.94495566s) [upload=452129]: Starting upload Dec 27 02:46:32.073950793 (515h17m31.945033838s) [upload=452129]: Done starting upload Dec 27 02:46:32.074094346 (515h17m31.945177382s) [upload=452129]: Uploading segments... Dec 27 02:46:32.074769092 (515h17m31.945852127s) [upload=452129]: Got next segment. Inline: false Dec 27 02:46:32.074803688 (515h17m31.945886723s) [upload=452129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:46:32.074808607 (515h17m31.945891652s) [upload=452129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:46:32.324070582 (515h17m32.195153617s) [seg_pos={0 0}, upload=452129]: Begin upload segment... Dec 27 02:46:32.324132040 (515h17m32.195215075s) [seg_pos={0 0}, upload=452129]: Done begin upload segment. Dec 27 02:46:32.324167487 (515h17m32.195250522s) [noise=true , address=45.91.116.14:4192 , piece_id=FTPMGKSUAGRM6JQFLL2S5L3RHBJ7HK6TWVBZUFKPQSUVDO3723HA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.324382005 (515h17m32.19546506s) [noise=true , address=145.79.227.11:4731 , piece_id=WHRW3YIP27B4P66B3SCL4CS23X6542RK7PIPCFYFER4PZXZ7TNXQ, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.325172401 (515h17m32.196255446s) [noise=true , address=145.79.227.19:4077 , piece_id=BMLXJSJALGYEZE3MOFC3TOTTWGOKUN3IMEBH2PAFSUD3JQ2COFXQ, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.325601888 (515h17m32.196684923s) [noise=true , address=145.79.227.50:4374 , piece_id=H7DQ7RDPFQFIPHWYMMQNYNJEYCZFNC2BEMZUGIMJJ4RMGCXMBG5Q, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.326072193 (515h17m32.197155228s) [noise=true , address=45.91.116.10:4082 , piece_id=7TFNRNZMQLIMWZOUOTWDBBXO4WUEW57ERNL5HKS6TIFQKN5K57OA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.326450033 (515h17m32.197533068s) [noise=true , address=145.79.227.42:4123 , piece_id=LPCYUTAMRHROYJHW3DJOI72RSEKGILFGLKCMEAB3HBKMVDTMHZFA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.326916912 (515h17m32.197999948s) [noise=true , address=77.90.31.11:4556 , piece_id=XNYN4IBUUOTYWTCHYSXMVRLWCAVED4QGIWR34FST3FFTVRLIE77Q, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.327245999 (515h17m32.198329034s) [noise=true , address=145.79.227.2:4115 , piece_id=AGDNBI7QLODD3EIXJIHSP24FPFEPTYG33YYUUC27KWL3WY3JRWOA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.327649057 (515h17m32.198732092s) [noise=true , address=145.79.227.14:4837 , piece_id=6J5JIPJIMXEFGKYBQ35HYID5Y5YOWE7UZTTY7QTWCMMLBDWYTYAA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.328900019 (515h17m32.199983064s) [noise=true , address=145.79.227.45:4229 , piece_id=XNGVRUMPRT3IA6ISSQTUPSSLN5FMPI7UMV5CNECSVZFMUBZGHIMA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.330752135 (515h17m32.20183517s) [noise=true , address=145.79.227.37:4567 , piece_id=5YZDEVRCETPIUKTD7E5OKA7SQWR7OZU5H6F47O4CNDM5TBYAOKGQ, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.330886802 (515h17m32.201969837s) [noise=true , address=145.79.227.13:4803 , piece_id=DN2SKC7SAVFDOVBKINI7W6UCBPKWZN5Q5CESMJWEDE7LK7NEY46Q, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.332517607 (515h17m32.203600642s) [noise=true , address=145.79.227.16:4912 , piece_id=P4QDVCF6YQQ36CSKAGBGVG44XL7LEFXNAO5FAA6NQKI2QH57IYZA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.333735597 (515h17m32.204818632s) [noise=true , address=45.91.116.7:4012 , piece_id=4QITOQWDSNVS4KFYBOB4HKPUVFB56WKC3MLVSOV5ATA3HIJIL4RA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.334984144 (515h17m32.206067179s) [noise=true , address=45.91.116.5:4298 , piece_id=XJKB6JBLXTSQ564NBVF4RCKU6BQU4ZD4YVPNNOOXJCJDCCZWHO2Q, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:32.336116860 (515h17m32.207199896s) [noise=true , address=145.79.227.3:4150 , piece_id=MJOK63MGIDXUEZIPYY6Y6HICZY4QPGANSJL6SGEGRETWUB43SECA, seg_pos={0 0}, upload=452129]: Uploading piece... Dec 27 02:46:33.955348416 (515h17m33.826431451s) [upload=452129]: Next returned nil segment Dec 27 02:46:33.955360699 (515h17m33.826443744s) [upload=452129]: Waiting for error group managing segments... Dec 27 02:46:34.326033374 (515h17m34.197116419s) [noise=true , address=145.79.227.11:4731 , piece_id=WHRW3YIP27B4P66B3SCL4CS23X6542RK7PIPCFYFER4PZXZ7TNXQ, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.327847779 (515h17m34.198930824s) [noise=true , address=145.79.227.14:4837 , piece_id=6J5JIPJIMXEFGKYBQ35HYID5Y5YOWE7UZTTY7QTWCMMLBDWYTYAA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.329466512 (515h17m34.200549547s) [noise=true , address=145.79.227.13:4803 , piece_id=DN2SKC7SAVFDOVBKINI7W6UCBPKWZN5Q5CESMJWEDE7LK7NEY46Q, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.330121459 (515h17m34.201204494s) [noise=true , address=145.79.227.2:4115 , piece_id=AGDNBI7QLODD3EIXJIHSP24FPFEPTYG33YYUUC27KWL3WY3JRWOA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.332050161 (515h17m34.203133206s) [noise=true , address=145.79.227.50:4374 , piece_id=H7DQ7RDPFQFIPHWYMMQNYNJEYCZFNC2BEMZUGIMJJ4RMGCXMBG5Q, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.332952920 (515h17m34.204035965s) [noise=true , address=145.79.227.16:4912 , piece_id=P4QDVCF6YQQ36CSKAGBGVG44XL7LEFXNAO5FAA6NQKI2QH57IYZA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.339286371 (515h17m34.210369406s) [noise=true , address=145.79.227.45:4229 , piece_id=XNGVRUMPRT3IA6ISSQTUPSSLN5FMPI7UMV5CNECSVZFMUBZGHIMA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.357791509 (515h17m34.228874544s) [noise=true , address=145.79.227.42:4123 , piece_id=LPCYUTAMRHROYJHW3DJOI72RSEKGILFGLKCMEAB3HBKMVDTMHZFA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.364033665 (515h17m34.2351167s) [noise=true , address=145.79.227.19:4077 , piece_id=BMLXJSJALGYEZE3MOFC3TOTTWGOKUN3IMEBH2PAFSUD3JQ2COFXQ, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.376069443 (515h17m34.247152478s) [noise=true , address=145.79.227.37:4567 , piece_id=5YZDEVRCETPIUKTD7E5OKA7SQWR7OZU5H6F47O4CNDM5TBYAOKGQ, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.561353201 (515h17m34.432436236s) [noise=true , address=45.91.116.5:4298 , piece_id=XJKB6JBLXTSQ564NBVF4RCKU6BQU4ZD4YVPNNOOXJCJDCCZWHO2Q, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.607361319 (515h17m34.478444354s) [noise=true , address=45.91.116.7:4012 , piece_id=4QITOQWDSNVS4KFYBOB4HKPUVFB56WKC3MLVSOV5ATA3HIJIL4RA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.608080429 (515h17m34.479163464s) [noise=true , address=77.90.31.11:4556 , piece_id=XNYN4IBUUOTYWTCHYSXMVRLWCAVED4QGIWR34FST3FFTVRLIE77Q, seg_pos={0 0}, upload=452129]: Done uploading piece. err: <nil> Dec 27 02:46:34.608089155 (515h17m34.47917219s) [seg_pos={0 0}, upload=452129]: Segment reached optimal threshold of 13 pieces. Dec 27 02:46:34.608260041 (515h17m34.479343076s) [noise=true , address=45.91.116.10:4082 , piece_id=7TFNRNZMQLIMWZOUOTWDBBXO4WUEW57ERNL5HKS6TIFQKN5K57OA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 02:46:34.608266703 (515h17m34.479349738s) [noise=true , address=45.91.116.10:4082 , piece_id=7TFNRNZMQLIMWZOUOTWDBBXO4WUEW57ERNL5HKS6TIFQKN5K57OA, seg_pos={0 0}, upload=452129]: Long tail cancellation due to optimal threshold Dec 27 02:46:34.608497593 (515h17m34.479580638s) [noise=true , address=45.91.116.14:4192 , piece_id=FTPMGKSUAGRM6JQFLL2S5L3RHBJ7HK6TWVBZUFKPQSUVDO3723HA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 02:46:34.608503975 (515h17m34.47958701s) [noise=true , address=45.91.116.14:4192 , piece_id=FTPMGKSUAGRM6JQFLL2S5L3RHBJ7HK6TWVBZUFKPQSUVDO3723HA, seg_pos={0 0}, upload=452129]: Long tail cancellation due to optimal threshold Dec 27 02:46:34.608598235 (515h17m34.47968127s) [noise=true , address=145.79.227.3:4150 , piece_id=MJOK63MGIDXUEZIPYY6Y6HICZY4QPGANSJL6SGEGRETWUB43SECA, seg_pos={0 0}, upload=452129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 27 02:46:34.608603164 (515h17m34.479686199s) [noise=true , address=145.79.227.3:4150 , piece_id=MJOK63MGIDXUEZIPYY6Y6HICZY4QPGANSJL6SGEGRETWUB43SECA, seg_pos={0 0}, upload=452129]: Long tail cancellation due to optimal threshold Dec 27 02:46:34.608616409 (515h17m34.479699444s) [seg_pos={0 0}, upload=452129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:46:34.608674981 (515h17m34.479758016s) [upload=452129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:46:34.608679119 (515h17m34.479762154s) [upload=452129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:46:34.897940290 (515h17m34.769023336s) [upload=452129]: Done uploading segments... Dec 27 02:46:34.897960249 (515h17m34.769043284s) [upload=452129]: Upload finished. err: <nil> Duration: 30.004481021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 28.990808661s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:46:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 28.990829371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.XX] duration: 30.003127728s trace: https://jaeger.dev.storj.io/trace/71435dcdb8340c95 FAIL (2025-12-27 02:45:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: SZ3GNQCUPYXAEW7AR3LQCOPPKQ4KSY7GG6HFXHJXT27EYA2SMKVA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: OGOFNWUIFAKIW4EIO4UJ2Z5RM2H2L7L6VZ46MMZHKBTKDLWOWTHQ) context deadline exceeded ======LOGS====== Dec 27 02:45:18.301640277 (515h16m18.172723313s) [upload=452116]: Starting upload Dec 27 02:45:18.301710992 (515h16m18.172794027s) [upload=452116]: Done starting upload Dec 27 02:45:18.301790022 (515h16m18.172873057s) [upload=452116]: Uploading segments... Dec 27 02:45:18.302031813 (515h16m18.173114848s) [upload=452116]: Got next segment. Inline: false Dec 27 02:45:18.302067451 (515h16m18.173150486s) [upload=452116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:45:18.302071949 (515h16m18.173154994s) [upload=452116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:45:18.547922629 (515h16m18.419005674s) [seg_pos={0 0}, upload=452116]: Begin upload segment... Dec 27 02:45:18.547982864 (515h16m18.419065899s) [seg_pos={0 0}, upload=452116]: Done begin upload segment. Dec 27 02:45:18.548024763 (515h16m18.419107798s) [noise=true , address=145.79.227.39:4013 , piece_id=UJERAGZBWIAFHZDNPP7D562NZQ5ESFRXWG3K6XBDIUPUN5MVWLIQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.548037637 (515h16m18.419120682s) [noise=true , address=145.79.227.29:4943 , piece_id=ASO37AHT4ONZ2U3MT3QRAHKTESHHUTWJMWWLU2SQVGCYCPVTWALA, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.548118001 (515h16m18.419201046s) [noise=true , address=145.79.227.2:4113 , piece_id=EP37ZSUYTI7P2HSEXPHMTTNTMAOU7DHNQPW75F6YWL5EKI3LY42A, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.548782116 (515h16m18.419865161s) [noise=true , address=145.79.227.38:4977 , piece_id=B5U636RCRY5ZUUNG5QSJCWV6IB6PAWUHXHCHMV5DDODCDQERK3UA, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.549126672 (515h16m18.420209717s) [noise=true , address=145.79.227.17:4972 , piece_id=6YMGJ64JCBXQDLJ4Z4WM7S6BUEUF3IDVQUD7KLSO62EUR35L3D3Q, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.549456770 (515h16m18.420539805s) [noise=true , address=145.79.227.52:4446 , piece_id=RPUFUB72M5SRL4VIZFLBZGRQKXPLWLAFDFRSPGQEG2RABRX62X7A, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.549479574 (515h16m18.420562619s) [noise=true , address=45.91.116.4:4937 , piece_id=HA3KHYH7EDN752XBU2FAFSTYWTHHIBMJXFUOCYHUKMO5UU7QD35Q, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.549900465 (515h16m18.42098351s) [noise=true , address=45.91.116.14:4191 , piece_id=SZ3GNQCUPYXAEW7AR3LQCOPPKQ4KSY7GG6HFXHJXT27EYA2SMKVA, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.550093352 (515h16m18.421176387s) [noise=true , address=145.79.227.42:4123 , piece_id=PJQ6UFUBFS6GXKJEMQCFKKH737DZFA2HQ5UVHC4725MNGOAVMHXQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.550187431 (515h16m18.421270466s) [noise=true , address=77.90.31.16:4736 , piece_id=OGOFNWUIFAKIW4EIO4UJ2Z5RM2H2L7L6VZ46MMZHKBTKDLWOWTHQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.550774118 (515h16m18.421857164s) [noise=true , address=145.79.227.47:4267 , piece_id=7TL2USLATL3RVS37UNRJUV7SD6TZWLM4XZWVKH6U4ACPS4GY4ATQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.551334405 (515h16m18.42241745s) [noise=true , address=145.79.227.11:4732 , piece_id=DUE7YUMQ3GR3HAD77XRPOSWRAAQQEOK7HGVJF367VQNKU7IZ6NTQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.551941191 (515h16m18.423024226s) [noise=true , address=145.79.227.36:4519 , piece_id=VP6WK6LHYARPO3PBLZNHHGC5VCHS443W7GN6DJZTEVKV2AIHAXUA, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.552504654 (515h16m18.423587689s) [noise=true , address=145.79.227.7:4293 , piece_id=YH3E5VZTPAXVK5LLWYG5YAOGMXLB2NLORYJIHYIDQXLCQWZJ53NA, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.553109565 (515h16m18.424192611s) [noise=true , address=45.91.116.6:4976 , piece_id=MMMYPCITJA76RNPRCW4OTAISMYUW2IQ4BGBNPHGUYRPPGD3PHYEQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:18.556778873 (515h16m18.427861918s) [noise=true , address=45.91.116.13:4154 , piece_id=F2TJWKZVQDFLYTIRUCPOQTB35VSNHLDZNGDSGZV4ZBXDBDKXPHMQ, seg_pos={0 0}, upload=452116]: Uploading piece... Dec 27 02:45:20.253968476 (515h16m20.125051511s) [upload=452116]: Next returned nil segment Dec 27 02:45:20.253990467 (515h16m20.125073503s) [upload=452116]: Waiting for error group managing segments... Dec 27 02:45:20.543483492 (515h16m20.414566537s) [noise=true , address=145.79.227.38:4977 , piece_id=B5U636RCRY5ZUUNG5QSJCWV6IB6PAWUHXHCHMV5DDODCDQERK3UA, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.546138066 (515h16m20.417221112s) [noise=true , address=145.79.227.7:4293 , piece_id=YH3E5VZTPAXVK5LLWYG5YAOGMXLB2NLORYJIHYIDQXLCQWZJ53NA, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.550754357 (515h16m20.421837392s) [noise=true , address=145.79.227.11:4732 , piece_id=DUE7YUMQ3GR3HAD77XRPOSWRAAQQEOK7HGVJF367VQNKU7IZ6NTQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.560593894 (515h16m20.431676929s) [noise=true , address=145.79.227.39:4013 , piece_id=UJERAGZBWIAFHZDNPP7D562NZQ5ESFRXWG3K6XBDIUPUN5MVWLIQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.566225978 (515h16m20.437309023s) [noise=true , address=145.79.227.2:4113 , piece_id=EP37ZSUYTI7P2HSEXPHMTTNTMAOU7DHNQPW75F6YWL5EKI3LY42A, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.567061058 (515h16m20.438144103s) [noise=true , address=145.79.227.52:4446 , piece_id=RPUFUB72M5SRL4VIZFLBZGRQKXPLWLAFDFRSPGQEG2RABRX62X7A, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.571789443 (515h16m20.442872488s) [noise=true , address=145.79.227.47:4267 , piece_id=7TL2USLATL3RVS37UNRJUV7SD6TZWLM4XZWVKH6U4ACPS4GY4ATQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.576767943 (515h16m20.447850988s) [noise=true , address=145.79.227.42:4123 , piece_id=PJQ6UFUBFS6GXKJEMQCFKKH737DZFA2HQ5UVHC4725MNGOAVMHXQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.584723211 (515h16m20.455806266s) [noise=true , address=145.79.227.17:4972 , piece_id=6YMGJ64JCBXQDLJ4Z4WM7S6BUEUF3IDVQUD7KLSO62EUR35L3D3Q, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.589420425 (515h16m20.46050347s) [noise=true , address=145.79.227.36:4519 , piece_id=VP6WK6LHYARPO3PBLZNHHGC5VCHS443W7GN6DJZTEVKV2AIHAXUA, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.806509186 (515h16m20.677592231s) [noise=true , address=45.91.116.13:4154 , piece_id=F2TJWKZVQDFLYTIRUCPOQTB35VSNHLDZNGDSGZV4ZBXDBDKXPHMQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:20.823804420 (515h16m20.694887455s) [noise=true , address=77.90.31.16:4736 , piece_id=OGOFNWUIFAKIW4EIO4UJ2Z5RM2H2L7L6VZ46MMZHKBTKDLWOWTHQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:21.543079696 (515h16m21.414162732s) [noise=true , address=45.91.116.14:4191 , piece_id=SZ3GNQCUPYXAEW7AR3LQCOPPKQ4KSY7GG6HFXHJXT27EYA2SMKVA, seg_pos={0 0}, upload=452116]: Done uploading piece. err: <nil> Dec 27 02:45:21.543117879 (515h16m21.414200924s) [seg_pos={0 0}, upload=452116]: Segment reached optimal threshold of 13 pieces. Dec 27 02:45:21.543373557 (515h16m21.414456592s) [noise=true , address=45.91.116.4:4937 , piece_id=HA3KHYH7EDN752XBU2FAFSTYWTHHIBMJXFUOCYHUKMO5UU7QD35Q, seg_pos={0 0}, upload=452116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 02:45:21.543383986 (515h16m21.414467031s) [noise=true , address=45.91.116.4:4937 , piece_id=HA3KHYH7EDN752XBU2FAFSTYWTHHIBMJXFUOCYHUKMO5UU7QD35Q, seg_pos={0 0}, upload=452116]: Long tail cancellation due to optimal threshold Dec 27 02:45:21.543505288 (515h16m21.414588323s) [noise=true , address=45.91.116.6:4976 , piece_id=MMMYPCITJA76RNPRCW4OTAISMYUW2IQ4BGBNPHGUYRPPGD3PHYEQ, seg_pos={0 0}, upload=452116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:57814->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 02:45:21.543556174 (515h16m21.414639239s) [noise=true , address=45.91.116.6:4976 , piece_id=MMMYPCITJA76RNPRCW4OTAISMYUW2IQ4BGBNPHGUYRPPGD3PHYEQ, seg_pos={0 0}, upload=452116]: Long tail cancellation due to optimal threshold Dec 27 02:45:21.543688216 (515h16m21.414771251s) [noise=true , address=145.79.227.29:4943 , piece_id=ASO37AHT4ONZ2U3MT3QRAHKTESHHUTWJMWWLU2SQVGCYCPVTWALA, seg_pos={0 0}, upload=452116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 02:45:21.543696311 (515h16m21.414779356s) [noise=true , address=145.79.227.29:4943 , piece_id=ASO37AHT4ONZ2U3MT3QRAHKTESHHUTWJMWWLU2SQVGCYCPVTWALA, seg_pos={0 0}, upload=452116]: Long tail cancellation due to optimal threshold Dec 27 02:45:21.543716279 (515h16m21.414799314s) [seg_pos={0 0}, upload=452116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:45:21.543786703 (515h16m21.414869738s) [upload=452116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:45:21.543792254 (515h16m21.414875299s) [upload=452116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:45:21.841300529 (515h16m21.712383585s) [upload=452116]: Done uploading segments... Dec 27 02:45:21.841323594 (515h16m21.712406629s) [upload=452116]: Upload finished. err: <nil> Duration: 30.00315018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 54.944602354s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:45:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 54.944632662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.X] duration: 30.002210169s trace: https://jaeger.dev.storj.io/trace/53878b06d34c5e6e FAIL (2025-12-27 02:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: MYNA7EYWQMQ4IV2MGYVYEHPG46TBHV7OF2SS5TKIUJCP2L2JRECQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: RO6OHFI4W6SR77ZQD5WGD5B6EWZJ2JS7M2TWUC6ZD2ZVE6MP6VJQ) context deadline exceeded ======LOGS====== ments... Dec 27 02:44:29.843708976 (515h15m29.714792021s) [upload=452101]: Got next segment. Inline: false Dec 27 02:44:29.843747409 (515h15m29.714830444s) [upload=452101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:44:29.843753732 (515h15m29.714836777s) [upload=452101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:44:30.091707105 (515h15m29.96279015s) [seg_pos={0 0}, upload=452101]: Begin upload segment... Dec 27 02:44:30.091787649 (515h15m29.962870694s) [seg_pos={0 0}, upload=452101]: Done begin upload segment. Dec 27 02:44:30.091814910 (515h15m29.962897945s) [noise=true , address=77.90.31.12:4589 , piece_id=RO6OHFI4W6SR77ZQD5WGD5B6EWZJ2JS7M2TWUC6ZD2ZVE6MP6VJQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.091824189 (515h15m29.962907224s) [noise=true , address=45.91.116.11:4119 , piece_id=E2OILMUV5YM5LS3YLCQVUI5P4YHXSS4FYXK7ODBY5GTR3JELXIUA, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.091897999 (515h15m29.962981034s) [noise=true , address=145.79.227.6:4259 , piece_id=2VVIZFCQNKNKFEINDNFPJCB2THGYRDED5RSQXDV27ZZT5VELMQYA, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.092304553 (515h15m29.963387588s) [noise=true , address=145.79.227.42:4123 , piece_id=EMIHAWQRT2X5EJZWVTIHXWQMLLGHTSIUFS5DWTTQNHWVXJX4VWOQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.092930805 (515h15m29.96401384s) [noise=true , address=145.79.227.13:4803 , piece_id=FXNWXQB7S7SY5YXN76TA5NMEYMILQCNGV4HJC7XWEJPTKXXF2LEQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.093406121 (515h15m29.964489156s) [noise=true , address=145.79.227.41:4085 , piece_id=C6OYBABU7SY34NRIUZSZAKESXYG4Y7UJZIXSFSY3G5B4TK7KZYXQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.093412823 (515h15m29.964495868s) [noise=true , address=45.91.116.7:4010 , piece_id=MYNA7EYWQMQ4IV2MGYVYEHPG46TBHV7OF2SS5TKIUJCP2L2JRECQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.093731740 (515h15m29.964814775s) [noise=true , address=145.79.227.45:4229 , piece_id=WZ2J7V463GW57M5N5W3Y6QXCQIUFNQ4A74AMM4OGIKNJVPWBAHNA, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.093771817 (515h15m29.964854853s) [noise=true , address=45.91.116.12:4225 , piece_id=4ZY2SKP4BZNMTW544WMS5UUYS3QBHLLASMM6DL5U3GSFELP52WCQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.094242694 (515h15m29.965325729s) [noise=true , address=145.79.227.11:4732 , piece_id=OVEJERGYRLO3SMQTIYB664VQWC6ZAKNI5ZE3VQUJZPV5WZKV7CHQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.094891489 (515h15m29.965974524s) [noise=true , address=45.91.116.14:4191 , piece_id=DSYKWLW6QZS33OSXMIFIB27SKYCP7GF2YEPM3FAERXJVM6WRXZ3Q, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.095341717 (515h15m29.966424752s) [noise=true , address=45.91.116.4:4939 , piece_id=BD6IMNHGY55OV7MYCL7JOP7ZECSYPT4RBOMZCCPVI3VYSTYW75GA, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.095926581 (515h15m29.967009626s) [noise=true , address=145.79.227.17:4972 , piece_id=GLC2PBH3YSW7HSASMJTUFQOQLHV46SCYUQHXLLSB5AU7OFIZX5SQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.096376156 (515h15m29.967459201s) [noise=true , address=45.91.116.5:4300 , piece_id=XJ6MX4GJBGW6NOQQ6WUJOAZE7LATYP24KSPJVFZNH4JGDP3TAMPA, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.096759706 (515h15m29.967842741s) [noise=true , address=145.79.227.48:4303 , piece_id=DNKJRBEHRNEPJBV5OTVXG6REX3QWAO6YGHNF6HXIJHIN25XF5C5A, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:30.097097119 (515h15m29.968180164s) [noise=true , address=145.79.227.50:4375 , piece_id=C24YH2HL4OW6F6WXW6POMJ3D7S5GLQAZRZP6HYXKVA2ZEQYEVCFQ, seg_pos={0 0}, upload=452101]: Uploading piece... Dec 27 02:44:31.803606379 (515h15m31.674689424s) [upload=452101]: Next returned nil segment Dec 27 02:44:31.803620275 (515h15m31.67470331s) [upload=452101]: Waiting for error group managing segments... Dec 27 02:44:32.085628756 (515h15m31.956711791s) [noise=true , address=145.79.227.17:4972 , piece_id=GLC2PBH3YSW7HSASMJTUFQOQLHV46SCYUQHXLLSB5AU7OFIZX5SQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.094795893 (515h15m31.965878958s) [noise=true , address=145.79.227.42:4123 , piece_id=EMIHAWQRT2X5EJZWVTIHXWQMLLGHTSIUFS5DWTTQNHWVXJX4VWOQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.098365129 (515h15m31.969448174s) [noise=true , address=145.79.227.50:4375 , piece_id=C24YH2HL4OW6F6WXW6POMJ3D7S5GLQAZRZP6HYXKVA2ZEQYEVCFQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.099872008 (515h15m31.970955043s) [noise=true , address=145.79.227.41:4085 , piece_id=C6OYBABU7SY34NRIUZSZAKESXYG4Y7UJZIXSFSY3G5B4TK7KZYXQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.102946533 (515h15m31.974029578s) [noise=true , address=145.79.227.6:4259 , piece_id=2VVIZFCQNKNKFEINDNFPJCB2THGYRDED5RSQXDV27ZZT5VELMQYA, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.103383875 (515h15m31.97446691s) [noise=true , address=145.79.227.13:4803 , piece_id=FXNWXQB7S7SY5YXN76TA5NMEYMILQCNGV4HJC7XWEJPTKXXF2LEQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.104293508 (515h15m31.975376553s) [noise=true , address=145.79.227.48:4303 , piece_id=DNKJRBEHRNEPJBV5OTVXG6REX3QWAO6YGHNF6HXIJHIN25XF5C5A, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.109975237 (515h15m31.981058272s) [noise=true , address=145.79.227.11:4732 , piece_id=OVEJERGYRLO3SMQTIYB664VQWC6ZAKNI5ZE3VQUJZPV5WZKV7CHQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.116363682 (515h15m31.987446747s) [noise=true , address=145.79.227.45:4229 , piece_id=WZ2J7V463GW57M5N5W3Y6QXCQIUFNQ4A74AMM4OGIKNJVPWBAHNA, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.327113503 (515h15m32.198196538s) [noise=true , address=77.90.31.12:4589 , piece_id=RO6OHFI4W6SR77ZQD5WGD5B6EWZJ2JS7M2TWUC6ZD2ZVE6MP6VJQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.354237062 (515h15m32.225320107s) [noise=true , address=45.91.116.7:4010 , piece_id=MYNA7EYWQMQ4IV2MGYVYEHPG46TBHV7OF2SS5TKIUJCP2L2JRECQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.388742306 (515h15m32.259825341s) [noise=true , address=45.91.116.4:4939 , piece_id=BD6IMNHGY55OV7MYCL7JOP7ZECSYPT4RBOMZCCPVI3VYSTYW75GA, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.452709532 (515h15m32.323792577s) [noise=true , address=45.91.116.5:4300 , piece_id=XJ6MX4GJBGW6NOQQ6WUJOAZE7LATYP24KSPJVFZNH4JGDP3TAMPA, seg_pos={0 0}, upload=452101]: Done uploading piece. err: <nil> Dec 27 02:44:32.452732436 (515h15m32.323815481s) [seg_pos={0 0}, upload=452101]: Segment reached optimal threshold of 13 pieces. Dec 27 02:44:32.452918019 (515h15m32.324001064s) [noise=true , address=45.91.116.14:4191 , piece_id=DSYKWLW6QZS33OSXMIFIB27SKYCP7GF2YEPM3FAERXJVM6WRXZ3Q, seg_pos={0 0}, upload=452101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 02:44:32.452924902 (515h15m32.324007947s) [noise=true , address=45.91.116.14:4191 , piece_id=DSYKWLW6QZS33OSXMIFIB27SKYCP7GF2YEPM3FAERXJVM6WRXZ3Q, seg_pos={0 0}, upload=452101]: Long tail cancellation due to optimal threshold Dec 27 02:44:32.453006649 (515h15m32.324089684s) [noise=true , address=45.91.116.11:4119 , piece_id=E2OILMUV5YM5LS3YLCQVUI5P4YHXSS4FYXK7ODBY5GTR3JELXIUA, seg_pos={0 0}, upload=452101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:55850->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:55850->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55850->45.91.116.11:4119: use of closed network connection Dec 27 02:44:32.453019362 (515h15m32.324102397s) [noise=true , address=45.91.116.11:4119 , piece_id=E2OILMUV5YM5LS3YLCQVUI5P4YHXSS4FYXK7ODBY5GTR3JELXIUA, seg_pos={0 0}, upload=452101]: Long tail cancellation due to optimal threshold Dec 27 02:44:32.453112690 (515h15m32.324195725s) [noise=true , address=45.91.116.12:4225 , piece_id=4ZY2SKP4BZNMTW544WMS5UUYS3QBHLLASMM6DL5U3GSFELP52WCQ, seg_pos={0 0}, upload=452101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:46372->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:46372->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46372->45.91.116.12:4225: use of closed network connection Dec 27 02:44:32.453118471 (515h15m32.324201516s) [noise=true , address=45.91.116.12:4225 , piece_id=4ZY2SKP4BZNMTW544WMS5UUYS3QBHLLASMM6DL5U3GSFELP52WCQ, seg_pos={0 0}, upload=452101]: Long tail cancellation due to optimal threshold Dec 27 02:44:32.453133279 (515h15m32.324216314s) [seg_pos={0 0}, upload=452101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:44:32.453194857 (515h15m32.324277902s) [upload=452101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:44:32.453199175 (515h15m32.32428221s) [upload=452101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:44:32.741689760 (515h15m32.612772795s) [upload=452101]: Done uploading segments... Dec 27 02:44:32.741708225 (515h15m32.61279126s) [upload=452101]: Upload finished. err: <nil> Duration: 30.002231909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 47.76955168s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 02:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 47.769579954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.XXX...X..X..X.X.XX.XXX.XX.XXXX.XX.XXXXX.X....XX.XXX.XXX.] duration: 6.769728295s trace: https://jaeger.dev.storj.io/trace/43ea7e62b0e4c719 FAIL (2025-12-27 02:43:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 02:43:10.709441419 (515h14m10.580524465s) [upload=452080]: Starting upload Dec 27 02:43:10.709526031 (515h14m10.580609076s) [upload=452080]: Done starting upload Dec 27 02:43:10.709653924 (515h14m10.580736969s) [upload=452080]: Uploading segments... Dec 27 02:43:10.709836312 (515h14m10.580919357s) [upload=452080]: Got next segment. Inline: false Dec 27 02:43:10.709891236 (515h14m10.580974281s) [upload=452080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 02:43:10.709897628 (515h14m10.580980683s) [upload=452080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 02:43:10.954917636 (515h14m10.826000691s) [seg_pos={0 0}, upload=452080]: Begin upload segment... Dec 27 02:43:10.954973212 (515h14m10.826056247s) [seg_pos={0 0}, upload=452080]: Done begin upload segment. Dec 27 02:43:10.955000935 (515h14m10.82608397s) [noise=true , address=45.91.116.14:4189 , piece_id=LHBNYWNJRVZHSKW5J6IAUXN2IS6QO3XEYNEN4W3CT6DY4HI52K2A, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.955012056 (515h14m10.826095101s) [noise=true , address=145.79.227.36:4517 , piece_id=QITECPEGRKYGOGKFMEDA2Z4NG7AZCN7BPXLCI5UNEYUGXL4UR7AQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.955389415 (515h14m10.82647245s) [noise=true , address=77.90.31.19:4844 , piece_id=O2BOZ3JZZCKA5BGFDJ3EYVNLKSFULFBNEXQFGOJVOVYI5LRVCTZQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.955682173 (515h14m10.826765218s) [noise=true , address=45.91.116.10:4082 , piece_id=UCUXDJMRLOMEU5NSTULYPONWHLPBODX6KKGUBE3EIVIKVS5GHLAQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.955942197 (515h14m10.827025232s) [noise=true , address=145.79.227.5:4222 , piece_id=N4EKHG7YFMOJYTEQQEVHAWZ4NY2BX53JABGDP5CPNYJZRPFFDVYQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.956269470 (515h14m10.827352505s) [noise=true , address=77.90.31.10:4519 , piece_id=BXLUS4JTXGD6FFZEK2JQEQCF7MYCECPEIV2VQDDRFKLWMMP2FFIA, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.956557840 (515h14m10.827640885s) [noise=true , address=145.79.227.37:4566 , piece_id=QUQ7HIWN2PCD3IDC5PB6HNJ44MNARNGAA77LK3DZGQMLI3OY7IBQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.956913236 (515h14m10.827996282s) [noise=true , address=145.79.227.14:4838 , piece_id=H74LZQLZLF2K7DA3FJPSLUFUDMAT2FMIJKQ2WLVQZE2N4ZCYBDIA, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.957381168 (515h14m10.828464203s) [noise=true , address=45.91.116.8:4047 , piece_id=VIS4AMDX6NE6WLORUXK24Y4DTKOZJKG37W3LKXTKJ7VNMDPFKMZQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.957739210 (515h14m10.828822255s) [noise=true , address=145.79.227.41:4085 , piece_id=VJ7R524QWAVR6W6KKDTNIMSP7VKZGIP3O3DBPRWG7SNUFP6M62KQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.958741959 (515h14m10.829824994s) [noise=true , address=145.79.227.49:4339 , piece_id=KHEDNH3VD4KGRZ32C4WUTEFIYO3T6SOA7U6B7MSCTMTJ3VLPJHCQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.959052390 (515h14m10.830135425s) [noise=true , address=45.91.116.12:4228 , piece_id=H7GTW4D3EKUV7QP7HGS43HUW2ZIF2NKL2FPOCTS5MCHBXNWP4YUQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.959889725 (515h14m10.83097276s) [noise=true , address=145.79.227.34:4905 , piece_id=XCBSCPMPSZGZP6AQQDAS5FBBYHQ42BKB2W5WV7NDFI2V65VP3AYA, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.960214192 (515h14m10.831297227s) [noise=true , address=145.79.227.48:4301 , piece_id=FSYOYRL3RAPYVPJTIQIYLVAKYUQ2LQATPMZK4CZT3EN55H4GB5TQ, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.961051116 (515h14m10.832134151s) [noise=true , address=145.79.227.46:4603 , piece_id=QD5G5XUGZO7LSY5ABUTIR5QVOXQGDS6TIM7VYZLAG5NJCR4FFBYA, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:10.962598994 (515h14m10.833682039s) [noise=true , address=45.91.116.7:4012 , piece_id=MXR3PUWSYNIGK3O4LWRSG76NR2WCLH7Z6VMPNGU5NBHGAJRYTEUA, seg_pos={0 0}, upload=452080]: Uploading piece... Dec 27 02:43:12.576177213 (515h14m12.447260258s) [upload=452080]: Next returned nil segment Dec 27 02:43:12.576218792 (515h14m12.447301837s) [upload=452080]: Waiting for error group managing segments... Dec 27 02:43:12.959648531 (515h14m12.830731566s) [noise=true , address=145.79.227.49:4339 , piece_id=KHEDNH3VD4KGRZ32C4WUTEFIYO3T6SOA7U6B7MSCTMTJ3VLPJHCQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:12.963950804 (515h14m12.835033849s) [noise=true , address=145.79.227.48:4301 , piece_id=FSYOYRL3RAPYVPJTIQIYLVAKYUQ2LQATPMZK4CZT3EN55H4GB5TQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:12.968786472 (515h14m12.839869507s) [noise=true , address=145.79.227.37:4566 , piece_id=QUQ7HIWN2PCD3IDC5PB6HNJ44MNARNGAA77LK3DZGQMLI3OY7IBQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:12.968843451 (515h14m12.839926496s) [noise=true , address=145.79.227.36:4517 , piece_id=QITECPEGRKYGOGKFMEDA2Z4NG7AZCN7BPXLCI5UNEYUGXL4UR7AQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:12.980609114 (515h14m12.851692149s) [noise=true , address=145.79.227.46:4603 , piece_id=QD5G5XUGZO7LSY5ABUTIR5QVOXQGDS6TIM7VYZLAG5NJCR4FFBYA, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:12.998076866 (515h14m12.869159901s) [noise=true , address=145.79.227.5:4222 , piece_id=N4EKHG7YFMOJYTEQQEVHAWZ4NY2BX53JABGDP5CPNYJZRPFFDVYQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.046881200 (515h14m12.917964235s) [noise=true , address=145.79.227.14:4838 , piece_id=H74LZQLZLF2K7DA3FJPSLUFUDMAT2FMIJKQ2WLVQZE2N4ZCYBDIA, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.058095965 (515h14m12.92917901s) [noise=true , address=145.79.227.41:4085 , piece_id=VJ7R524QWAVR6W6KKDTNIMSP7VKZGIP3O3DBPRWG7SNUFP6M62KQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.253460299 (515h14m13.124543344s) [noise=true , address=77.90.31.19:4844 , piece_id=O2BOZ3JZZCKA5BGFDJ3EYVNLKSFULFBNEXQFGOJVOVYI5LRVCTZQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.278083366 (515h14m13.149166411s) [noise=true , address=45.91.116.8:4047 , piece_id=VIS4AMDX6NE6WLORUXK24Y4DTKOZJKG37W3LKXTKJ7VNMDPFKMZQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.318356133 (515h14m13.189439178s) [noise=true , address=45.91.116.14:4189 , piece_id=LHBNYWNJRVZHSKW5J6IAUXN2IS6QO3XEYNEN4W3CT6DY4HI52K2A, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.867376500 (515h14m13.738459545s) [noise=true , address=45.91.116.7:4012 , piece_id=MXR3PUWSYNIGK3O4LWRSG76NR2WCLH7Z6VMPNGU5NBHGAJRYTEUA, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.956359963 (515h14m13.827442998s) [noise=true , address=45.91.116.10:4082 , piece_id=UCUXDJMRLOMEU5NSTULYPONWHLPBODX6KKGUBE3EIVIKVS5GHLAQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: <nil> Dec 27 02:43:13.956382536 (515h14m13.827465581s) [seg_pos={0 0}, upload=452080]: Segment reached optimal threshold of 13 pieces. Dec 27 02:43:13.956610330 (515h14m13.827693375s) [noise=true , address=77.90.31.10:4519 , piece_id=BXLUS4JTXGD6FFZEK2JQEQCF7MYCECPEIV2VQDDRFKLWMMP2FFIA, seg_pos={0 0}, upload=452080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 02:43:13.956619457 (515h14m13.827702502s) [noise=true , address=77.90.31.10:4519 , piece_id=BXLUS4JTXGD6FFZEK2JQEQCF7MYCECPEIV2VQDDRFKLWMMP2FFIA, seg_pos={0 0}, upload=452080]: Long tail cancellation due to optimal threshold Dec 27 02:43:13.956827854 (515h14m13.827910899s) [noise=true , address=145.79.227.34:4905 , piece_id=XCBSCPMPSZGZP6AQQDAS5FBBYHQ42BKB2W5WV7NDFI2V65VP3AYA, seg_pos={0 0}, upload=452080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 02:43:13.956856398 (515h14m13.827939463s) [noise=true , address=145.79.227.34:4905 , piece_id=XCBSCPMPSZGZP6AQQDAS5FBBYHQ42BKB2W5WV7NDFI2V65VP3AYA, seg_pos={0 0}, upload=452080]: Long tail cancellation due to optimal threshold Dec 27 02:43:13.956918717 (515h14m13.828001752s) [noise=true , address=45.91.116.12:4228 , piece_id=H7GTW4D3EKUV7QP7HGS43HUW2ZIF2NKL2FPOCTS5MCHBXNWP4YUQ, seg_pos={0 0}, upload=452080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 02:43:13.956926923 (515h14m13.828009958s) [noise=true , address=45.91.116.12:4228 , piece_id=H7GTW4D3EKUV7QP7HGS43HUW2ZIF2NKL2FPOCTS5MCHBXNWP4YUQ, seg_pos={0 0}, upload=452080]: Long tail cancellation due to optimal threshold Dec 27 02:43:13.956951339 (515h14m13.828034384s) [seg_pos={0 0}, upload=452080]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 02:43:13.957019067 (515h14m13.828102102s) [upload=452080]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:43:13.957024689 (515h14m13.828107724s) [upload=452080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:43:14.258365062 (515h14m14.129448098s) [upload=452080]: Done uploading segments... Dec 27 02:43:14.258384840 (515h14m14.129467885s) [upload=452080]: Upload finished. err: <nil> Duration: 6.769751299s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)