Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..................X.X................................X.....] duration: 30.831363353s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:23:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 30.83138805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X...XXX..X..X........X...XX..........XX.X..X.XX....XXX..] duration: 5.742881635s trace: https://jaeger.dev.storj.io/trace/24749fc9e82a8066 FAIL (2025-12-31 19:22:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:21:30.716797328 (627h52m30.587880364s) [upload=550187]: Starting upload Dec 31 19:21:30.716915113 (627h52m30.587998148s) [upload=550187]: Done starting upload Dec 31 19:21:30.717042355 (627h52m30.58812539s) [upload=550187]: Uploading segments... Dec 31 19:21:30.717184656 (627h52m30.588267691s) [upload=550187]: Got next segment. Inline: false Dec 31 19:21:30.717216617 (627h52m30.588299652s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:21:30.717221817 (627h52m30.588304852s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:21:30.965318716 (627h52m30.836401771s) [seg_pos={0 0}, upload=550187]: Begin upload segment... Dec 31 19:21:30.965377467 (627h52m30.836460502s) [seg_pos={0 0}, upload=550187]: Done begin upload segment. Dec 31 19:21:30.965415009 (627h52m30.836498054s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.965436520 (627h52m30.836519555s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966030431 (627h52m30.837113466s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966037734 (627h52m30.837120779s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966435402 (627h52m30.837518437s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966481800 (627h52m30.837564835s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966866583 (627h52m30.837949618s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966876852 (627h52m30.837959887s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967231637 (627h52m30.838314673s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967277766 (627h52m30.838360801s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967648461 (627h52m30.838731506s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968113087 (627h52m30.839196122s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968486557 (627h52m30.839569592s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.969129782 (627h52m30.840212817s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.970001332 (627h52m30.841084367s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.972402444 (627h52m30.843485479s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:32.662428143 (627h52m32.533511198s) [upload=550187]: Next returned nil segment Dec 31 19:21:32.662444755 (627h52m32.5335278s) [upload=550187]: Waiting for error group managing segments... Dec 31 19:21:32.945449384 (627h52m32.816532429s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.949760072 (627h52m32.820843117s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.954130464 (627h52m32.825213509s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.955825852 (627h52m32.826908898s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.962706174 (627h52m32.833789209s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.972171968 (627h52m32.843255053s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.991907246 (627h52m32.862990281s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.138253609 (627h52m33.009336644s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.229063945 (627h52m33.10014698s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.325420511 (627h52m33.196503556s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.341958131 (627h52m33.213041166s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.374487280 (627h52m33.245570325s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401736056 (627h52m33.272819101s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401757126 (627h52m33.272840172s) [seg_pos={0 0}, upload=550187]: Segment reached optimal threshold of 13 pieces. Dec 31 19:21:33.401966936 (627h52m33.273049971s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.401974761 (627h52m33.273057796s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402120017 (627h52m33.273203062s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402128213 (627h52m33.273211258s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402226951 (627h52m33.273309986s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402258451 (627h52m33.273341516s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402310039 (627h52m33.273393074s) [seg_pos={0 0}, upload=550187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:21:33.402411863 (627h52m33.273494908s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:21:33.402419979 (627h52m33.273503034s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:21:33.691054349 (627h52m33.562137394s) [upload=550187]: Done uploading segments... Dec 31 19:21:33.691072173 (627h52m33.562155208s) [upload=550187]: Upload finished. err: <nil> Duration: 5.742908877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.X...XXX..X..X........X...XX..........XX.X..X.XX....XXX.] duration: 17.472287053s trace: https://jaeger.dev.storj.io/trace/24749fc9e82a8066 FAIL (2025-12-31 19:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:21:30.716797328 (627h52m30.587880364s) [upload=550187]: Starting upload Dec 31 19:21:30.716915113 (627h52m30.587998148s) [upload=550187]: Done starting upload Dec 31 19:21:30.717042355 (627h52m30.58812539s) [upload=550187]: Uploading segments... Dec 31 19:21:30.717184656 (627h52m30.588267691s) [upload=550187]: Got next segment. Inline: false Dec 31 19:21:30.717216617 (627h52m30.588299652s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:21:30.717221817 (627h52m30.588304852s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:21:30.965318716 (627h52m30.836401771s) [seg_pos={0 0}, upload=550187]: Begin upload segment... Dec 31 19:21:30.965377467 (627h52m30.836460502s) [seg_pos={0 0}, upload=550187]: Done begin upload segment. Dec 31 19:21:30.965415009 (627h52m30.836498054s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.965436520 (627h52m30.836519555s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966030431 (627h52m30.837113466s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966037734 (627h52m30.837120779s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966435402 (627h52m30.837518437s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966481800 (627h52m30.837564835s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966866583 (627h52m30.837949618s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966876852 (627h52m30.837959887s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967231637 (627h52m30.838314673s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967277766 (627h52m30.838360801s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967648461 (627h52m30.838731506s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968113087 (627h52m30.839196122s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968486557 (627h52m30.839569592s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.969129782 (627h52m30.840212817s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.970001332 (627h52m30.841084367s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.972402444 (627h52m30.843485479s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:32.662428143 (627h52m32.533511198s) [upload=550187]: Next returned nil segment Dec 31 19:21:32.662444755 (627h52m32.5335278s) [upload=550187]: Waiting for error group managing segments... Dec 31 19:21:32.945449384 (627h52m32.816532429s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.949760072 (627h52m32.820843117s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.954130464 (627h52m32.825213509s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.955825852 (627h52m32.826908898s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.962706174 (627h52m32.833789209s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.972171968 (627h52m32.843255053s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.991907246 (627h52m32.862990281s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.138253609 (627h52m33.009336644s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.229063945 (627h52m33.10014698s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.325420511 (627h52m33.196503556s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.341958131 (627h52m33.213041166s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.374487280 (627h52m33.245570325s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401736056 (627h52m33.272819101s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401757126 (627h52m33.272840172s) [seg_pos={0 0}, upload=550187]: Segment reached optimal threshold of 13 pieces. Dec 31 19:21:33.401966936 (627h52m33.273049971s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.401974761 (627h52m33.273057796s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402120017 (627h52m33.273203062s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402128213 (627h52m33.273211258s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402226951 (627h52m33.273309986s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402258451 (627h52m33.273341516s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402310039 (627h52m33.273393074s) [seg_pos={0 0}, upload=550187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:21:33.402411863 (627h52m33.273494908s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:21:33.402419979 (627h52m33.273503034s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:21:33.691054349 (627h52m33.562137394s) [upload=550187]: Done uploading segments... Dec 31 19:21:33.691072173 (627h52m33.562155208s) [upload=550187]: Upload finished. err: <nil> Duration: 17.472303293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..................X.X................................X....] duration: 15.892100753s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:22:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 15.892125359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...X.X...XXX..X..X........X...XX..........XX.X..X.XX....XXX] duration: 30.003749819s trace: https://jaeger.dev.storj.io/trace/24749fc9e82a8066 FAIL (2025-12-31 19:22:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:21:30.716797328 (627h52m30.587880364s) [upload=550187]: Starting upload Dec 31 19:21:30.716915113 (627h52m30.587998148s) [upload=550187]: Done starting upload Dec 31 19:21:30.717042355 (627h52m30.58812539s) [upload=550187]: Uploading segments... Dec 31 19:21:30.717184656 (627h52m30.588267691s) [upload=550187]: Got next segment. Inline: false Dec 31 19:21:30.717216617 (627h52m30.588299652s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:21:30.717221817 (627h52m30.588304852s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:21:30.965318716 (627h52m30.836401771s) [seg_pos={0 0}, upload=550187]: Begin upload segment... Dec 31 19:21:30.965377467 (627h52m30.836460502s) [seg_pos={0 0}, upload=550187]: Done begin upload segment. Dec 31 19:21:30.965415009 (627h52m30.836498054s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.965436520 (627h52m30.836519555s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966030431 (627h52m30.837113466s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966037734 (627h52m30.837120779s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966435402 (627h52m30.837518437s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966481800 (627h52m30.837564835s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966866583 (627h52m30.837949618s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.966876852 (627h52m30.837959887s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967231637 (627h52m30.838314673s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967277766 (627h52m30.838360801s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.967648461 (627h52m30.838731506s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968113087 (627h52m30.839196122s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.968486557 (627h52m30.839569592s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.969129782 (627h52m30.840212817s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.970001332 (627h52m30.841084367s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:30.972402444 (627h52m30.843485479s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Uploading piece... Dec 31 19:21:32.662428143 (627h52m32.533511198s) [upload=550187]: Next returned nil segment Dec 31 19:21:32.662444755 (627h52m32.5335278s) [upload=550187]: Waiting for error group managing segments... Dec 31 19:21:32.945449384 (627h52m32.816532429s) [noise=true , address=145.79.227.8:4330 , piece_id=AYT3QFE7S5IQ6NV66AV3Y6NRBY5EWYZ6BACST7CJ2VB57ZKNJMWQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.949760072 (627h52m32.820843117s) [noise=true , address=145.79.227.51:4410 , piece_id=H42KKZYWST55HEVXG4WQYPN4NGVLVNXKOK2D2VUAQOEA7OLHNFNQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.954130464 (627h52m32.825213509s) [noise=true , address=145.79.227.9:4366 , piece_id=MIAA4CVP6VOEFBVVWHQ4B5SMVHSXXLLN2RA4EHYLXI5HNGD2LT3Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.955825852 (627h52m32.826908898s) [noise=true , address=145.79.227.2:4113 , piece_id=OJMYRZ6T3M7W3F5ITUM4CNJHK7PNSGQFNNAI2K5MDVDWBJRFCTCQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.962706174 (627h52m32.833789209s) [noise=true , address=145.79.227.19:4077 , piece_id=EH4ULZIP7AKWY32MIAMQZIATXGXRIQVIPLUIVNZGZJRAWYITDX2Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.972171968 (627h52m32.843255053s) [noise=true , address=145.79.227.38:4979 , piece_id=T2VCGG5HFZUTWWQGWNP5GITCIDE6FLAWQSYNCZYZGFSXVTB343HQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:32.991907246 (627h52m32.862990281s) [noise=true , address=145.79.227.31:4798 , piece_id=ZIZ6B63FYSFZAFA2KMNVLHDATIRXAOUTP744NLLCXTGR2LGM7VGA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.138253609 (627h52m33.009336644s) [noise=true , address=77.90.31.19:4841 , piece_id=TZNWMWS5N2OS4UPIMXRIFKKQQ4VPWM76OYZZHPJ4PBBPV2J75Y5A, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.229063945 (627h52m33.10014698s) [noise=true , address=77.90.31.15:4700 , piece_id=VBNO5TYOVNNKO3NH4OLOGH2Q4Y775EOZDFFUCWXG63BREPSDAWMA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.325420511 (627h52m33.196503556s) [noise=true , address=45.91.116.8:4046 , piece_id=UD4K5L3Z7P2WXWSKBLFM6FFV64QSZBCDC45UBKGAMDXMMAEQKVMQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.341958131 (627h52m33.213041166s) [noise=true , address=45.91.116.4:4940 , piece_id=UK4VWU5HM7FAGDW4Q36EI5OZBJFM4W6WUOK2MV2I7KPFNX7HLFIQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.374487280 (627h52m33.245570325s) [noise=true , address=45.91.116.9:4263 , piece_id=GEN5J3J7IHS7CDJSFVCRNWIXK6I33NNNLJTDO4TL5K7E4R42K4UQ, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401736056 (627h52m33.272819101s) [noise=true , address=45.91.116.12:4227 , piece_id=ZAJ6E5ZFJFXQWKVZZKFZWU7QXFHW3PBAXYEWIWCOG6WM363KZPVA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: <nil> Dec 31 19:21:33.401757126 (627h52m33.272840172s) [seg_pos={0 0}, upload=550187]: Segment reached optimal threshold of 13 pieces. Dec 31 19:21:33.401966936 (627h52m33.273049971s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.401974761 (627h52m33.273057796s) [noise=true , address=45.91.116.7:4012 , piece_id=2GBWOB2R2DPCMIBMO5SHPTZ4EC2V3745KZWWXZ4IIBFQSGS2374Q, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402120017 (627h52m33.273203062s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402128213 (627h52m33.273211258s) [noise=true , address=145.79.227.30:4762 , piece_id=GQR3FAUKWCD3A4JFL4BB2VY4MNOSMXDSITIG5LZWMFXHTJBRCSPA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402226951 (627h52m33.273309986s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 31 19:21:33.402258451 (627h52m33.273341516s) [noise=true , address=145.79.227.33:4869 , piece_id=4NKY2ROZUI7ZW3SIAHLSCFPEABL3PFK2W72ITLHGJXNQBZJI4THA, seg_pos={0 0}, upload=550187]: Long tail cancellation due to optimal threshold Dec 31 19:21:33.402310039 (627h52m33.273393074s) [seg_pos={0 0}, upload=550187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:21:33.402411863 (627h52m33.273494908s) [upload=550187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:21:33.402419979 (627h52m33.273503034s) [upload=550187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:21:33.691054349 (627h52m33.562137394s) [upload=550187]: Done uploading segments... Dec 31 19:21:33.691072173 (627h52m33.562155208s) [upload=550187]: Upload finished. err: <nil> Duration: 30.00377632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..................X.X................................X...] duration: 25.549093001s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 25.549122496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...X.X...XXX..X..X........X...XX..........XX.X..X.XX....XX] duration: 30.003564269s trace: https://jaeger.dev.storj.io/trace/18c8f56a4ed7f5d5 FAIL (2025-12-31 19:20:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4RHMIPBYUIYQX3U5DIMLJ677WSEVKUMBPBQNJR3CO4TWOZSYB5JA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: UAM4ZRD3AKBWBDOWEGD2O5ADJRYPV6256KVEGFEWDBLRJGVQ7NJQ) context deadline exceeded ======LOGS====== 662037189 (627h51m18.533120224s) [upload=550170]: Uploading segments... Dec 31 19:20:18.662123213 (627h51m18.533206258s) [upload=550170]: Got next segment. Inline: false Dec 31 19:20:18.662150385 (627h51m18.53323342s) [upload=550170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:20:18.662154753 (627h51m18.533237798s) [upload=550170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:20:18.908347507 (627h51m18.779430552s) [seg_pos={0 0}, upload=550170]: Begin upload segment... Dec 31 19:20:18.908402181 (627h51m18.779485216s) [seg_pos={0 0}, upload=550170]: Done begin upload segment. Dec 31 19:20:18.908427168 (627h51m18.779510203s) [noise=true , address=145.79.227.47:4266 , piece_id=DAC74GBZ3HPOIBTHZEZCMDQURBPJ6Z2KR7EOXUUXU4Z6M3C4QNFQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.908438119 (627h51m18.779521164s) [noise=true , address=45.91.116.9:4264 , piece_id=7F7CIFM2B4FDDFQO5PI2DM7OR34CT5L3VS2MBFXZRD4EMKIMICEA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.909058480 (627h51m18.780141525s) [noise=true , address=145.79.227.3:4151 , piece_id=J6RXQCOUXGSV3S5RED2KXHMAH74GDFO6XMPO66WHCKINWDFGPHAQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.909487959 (627h51m18.780570994s) [noise=true , address=145.79.227.49:4339 , piece_id=FFNBIPDEVCV7ZZPBSXSXFKREHG6BLQZZEG65VXCWM6TNNAXDF2SQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.910024921 (627h51m18.781107966s) [noise=true , address=145.79.227.42:4121 , piece_id=XMKRGOU7QCEIKEUSSG6RAMQXEKNZ5E2DGE3QVEDB2RQ2ONTNZXLQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.910422277 (627h51m18.781505322s) [noise=true , address=45.91.116.13:4154 , piece_id=NPSXWLT5KCJW2X65DCQWQWSN5U2QMY6YJADFFIOX3H52CHIGHGUA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.911780935 (627h51m18.78286398s) [noise=true , address=77.90.31.10:4517 , piece_id=LMP57SIPOGFZMH5IK5YVBWQ5QM4V5R5RHEMH7CMMN2JUU2ZTHAZA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.912183922 (627h51m18.783266957s) [noise=true , address=77.90.31.14:4664 , piece_id=NZ4DISSLR3ID4PVM7AHFBSIPOEOZ3XKXOCQQE3M2HZ2SBTOUA7AQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.913690881 (627h51m18.784773926s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMIPBYUIYQX3U5DIMLJ677WSEVKUMBPBQNJR3CO4TWOZSYB5JA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.916932644 (627h51m18.788015689s) [noise=true , address=45.91.116.14:4191 , piece_id=UAM4ZRD3AKBWBDOWEGD2O5ADJRYPV6256KVEGFEWDBLRJGVQ7NJQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.920422038 (627h51m18.791505083s) [noise=true , address=77.90.31.11:4555 , piece_id=6FPCB4H4L55ES3BMM5622ZEZGD37F3BFKLVAEZ56H66X44QHI2PA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.923452469 (627h51m18.794535514s) [noise=true , address=145.79.227.38:4978 , piece_id=3MNSGEOD52IZDD5VXTC4SDCET3RKT7A6D4DNCTTUDFI2OK7KMJOA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.924999575 (627h51m18.79608262s) [noise=true , address=145.79.227.19:4078 , piece_id=ZU4SUCGBDHVCGR2CRP622UOQ436JKMQN75TQDNKLD6GCPWN3AQKA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.928136528 (627h51m18.799219573s) [noise=true , address=145.79.227.2:4114 , piece_id=WZS2BXRNPIDDS7JY4TH2JSM3NG5PSDI4P66BJ4JOEKRXN5XECKPA, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.930298836 (627h51m18.801381871s) [noise=true , address=45.91.116.12:4228 , piece_id=A2S2LP3ZGAIVKXJMACDHQGMEFTY44YD56UFPB4LW5GRNC2GV6JFQ, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:18.933025317 (627h51m18.804108362s) [noise=true , address=145.79.227.36:4518 , piece_id=FCRIZY2A3GPQE3HM3NZ6UHRCODLI2ZQJZOV4SMLIXO2RWCC5T27Q, seg_pos={0 0}, upload=550170]: Uploading piece... Dec 31 19:20:20.604330219 (627h51m20.475413264s) [upload=550170]: Next returned nil segment Dec 31 19:20:20.604344936 (627h51m20.475427982s) [upload=550170]: Waiting for error group managing segments... Dec 31 19:20:20.901664864 (627h51m20.772747909s) [noise=true , address=145.79.227.3:4151 , piece_id=J6RXQCOUXGSV3S5RED2KXHMAH74GDFO6XMPO66WHCKINWDFGPHAQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:20.901735719 (627h51m20.772818764s) [noise=true , address=145.79.227.42:4121 , piece_id=XMKRGOU7QCEIKEUSSG6RAMQXEKNZ5E2DGE3QVEDB2RQ2ONTNZXLQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:20.906243242 (627h51m20.777326287s) [noise=true , address=145.79.227.36:4518 , piece_id=FCRIZY2A3GPQE3HM3NZ6UHRCODLI2ZQJZOV4SMLIXO2RWCC5T27Q, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:20.913105087 (627h51m20.784188122s) [noise=true , address=145.79.227.38:4978 , piece_id=3MNSGEOD52IZDD5VXTC4SDCET3RKT7A6D4DNCTTUDFI2OK7KMJOA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:20.931354257 (627h51m20.802437302s) [noise=true , address=145.79.227.2:4114 , piece_id=WZS2BXRNPIDDS7JY4TH2JSM3NG5PSDI4P66BJ4JOEKRXN5XECKPA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:20.983951790 (627h51m20.855034835s) [noise=true , address=145.79.227.47:4266 , piece_id=DAC74GBZ3HPOIBTHZEZCMDQURBPJ6Z2KR7EOXUUXU4Z6M3C4QNFQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.167768913 (627h51m21.038851948s) [noise=true , address=77.90.31.14:4664 , piece_id=NZ4DISSLR3ID4PVM7AHFBSIPOEOZ3XKXOCQQE3M2HZ2SBTOUA7AQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.203178997 (627h51m21.074262042s) [noise=true , address=77.90.31.10:4517 , piece_id=LMP57SIPOGFZMH5IK5YVBWQ5QM4V5R5RHEMH7CMMN2JUU2ZTHAZA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.231510324 (627h51m21.102593359s) [noise=true , address=45.91.116.6:4975 , piece_id=4RHMIPBYUIYQX3U5DIMLJ677WSEVKUMBPBQNJR3CO4TWOZSYB5JA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.282464137 (627h51m21.153547172s) [noise=true , address=45.91.116.14:4191 , piece_id=UAM4ZRD3AKBWBDOWEGD2O5ADJRYPV6256KVEGFEWDBLRJGVQ7NJQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.307215769 (627h51m21.178298814s) [noise=true , address=45.91.116.12:4228 , piece_id=A2S2LP3ZGAIVKXJMACDHQGMEFTY44YD56UFPB4LW5GRNC2GV6JFQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.346023363 (627h51m21.217106408s) [noise=true , address=77.90.31.11:4555 , piece_id=6FPCB4H4L55ES3BMM5622ZEZGD37F3BFKLVAEZ56H66X44QHI2PA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.989680944 (627h51m21.860763979s) [noise=true , address=45.91.116.9:4264 , piece_id=7F7CIFM2B4FDDFQO5PI2DM7OR34CT5L3VS2MBFXZRD4EMKIMICEA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: <nil> Dec 31 19:20:21.989706994 (627h51m21.860790039s) [seg_pos={0 0}, upload=550170]: Segment reached optimal threshold of 13 pieces. Dec 31 19:20:21.989966929 (627h51m21.861049974s) [noise=true , address=45.91.116.13:4154 , piece_id=NPSXWLT5KCJW2X65DCQWQWSN5U2QMY6YJADFFIOX3H52CHIGHGUA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:40106->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:40106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40106->45.91.116.13:4154: use of closed network connection Dec 31 19:20:21.989976948 (627h51m21.861060024s) [noise=true , address=45.91.116.13:4154 , piece_id=NPSXWLT5KCJW2X65DCQWQWSN5U2QMY6YJADFFIOX3H52CHIGHGUA, seg_pos={0 0}, upload=550170]: Long tail cancellation due to optimal threshold Dec 31 19:20:21.990005924 (627h51m21.861088969s) [noise=true , address=145.79.227.49:4339 , piece_id=FFNBIPDEVCV7ZZPBSXSXFKREHG6BLQZZEG65VXCWM6TNNAXDF2SQ, seg_pos={0 0}, upload=550170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:20:21.990018708 (627h51m21.861101753s) [noise=true , address=145.79.227.49:4339 , piece_id=FFNBIPDEVCV7ZZPBSXSXFKREHG6BLQZZEG65VXCWM6TNNAXDF2SQ, seg_pos={0 0}, upload=550170]: Long tail cancellation due to optimal threshold Dec 31 19:20:21.990292459 (627h51m21.861375504s) [noise=true , address=145.79.227.19:4078 , piece_id=ZU4SUCGBDHVCGR2CRP622UOQ436JKMQN75TQDNKLD6GCPWN3AQKA, seg_pos={0 0}, upload=550170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:38568->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:38568->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: context canceled Dec 31 19:20:21.990301827 (627h51m21.861384872s) [noise=true , address=145.79.227.19:4078 , piece_id=ZU4SUCGBDHVCGR2CRP622UOQ436JKMQN75TQDNKLD6GCPWN3AQKA, seg_pos={0 0}, upload=550170]: Long tail cancellation due to optimal threshold Dec 31 19:20:21.990343937 (627h51m21.861426982s) [seg_pos={0 0}, upload=550170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:20:21.990461511 (627h51m21.861544546s) [upload=550170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:20:21.990468494 (627h51m21.861551539s) [upload=550170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:20:22.282200770 (627h51m22.153283815s) [upload=550170]: Done uploading segments... Dec 31 19:20:22.282225267 (627h51m22.153308302s) [upload=550170]: Upload finished. err: <nil> Duration: 30.00358617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..................X.X................................X..] duration: 30.986604509s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 30.986634024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X...X.X...XXX..X..X........X...XX..........XX.X..X.XX....X] duration: 30.002131845s trace: https://jaeger.dev.storj.io/trace/7ecf6d5eecfde9fc FAIL (2025-12-31 19:19:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 31 19:19:22.743947809 (627h50m22.615030854s) [upload=550154]: Got next segment. Inline: false Dec 31 19:19:22.743996351 (627h50m22.615079396s) [upload=550154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:19:22.744002222 (627h50m22.615085257s) [upload=550154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:19:22.992004111 (627h50m22.863087146s) [seg_pos={0 0}, upload=550154]: Begin upload segment... Dec 31 19:19:22.992078022 (627h50m22.863161067s) [seg_pos={0 0}, upload=550154]: Done begin upload segment. Dec 31 19:19:22.992134799 (627h50m22.863217834s) [noise=true , address=145.79.227.14:4838 , piece_id=RO74PIBDAVLNGO2FOX3WXJNCWRTT2VCXVMVDG5GJSBYN66YUGUSQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.992183442 (627h50m22.863266477s) [noise=true , address=145.79.227.50:4375 , piece_id=36HZINMYQ7NJNWA7RZXAGPIDPEE5FSBPMAOM6OVINFIYIDSU3XWQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.992715154 (627h50m22.8637982s) [noise=true , address=145.79.227.23:4546 , piece_id=4LGWC7FMB2GPFU3ED2KGB3DRVJNRRDERBAPUAQNRMDAEYYMD2WZQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.993204937 (627h50m22.864287982s) [noise=true , address=77.90.31.10:4520 , piece_id=E7KCAIUBEDLFUB3O7RESBSASNR6SBU3P62P6Y7LWKNGL6VY7PLLA, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.993554452 (627h50m22.864637488s) [noise=true , address=145.79.227.37:4566 , piece_id=ELNTCOKPH2XDI4KWGHWISBZVEU44WEII2XGAZC3U6A4ICR65QU3A, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.993890613 (627h50m22.864973658s) [noise=true , address=45.91.116.10:4081 , piece_id=GUHH5D6TIOPULXOVF6M2R3JWV7AHIYFIGHRF42PTHY7HEW4AVL6Q, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.993995613 (627h50m22.865078658s) [noise=true , address=145.79.227.2:4114 , piece_id=KW65HIVKPRVWH7VBA2KCBDAHVTXVQZQAXV567ORQI56UYMBVOEBQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.994164364 (627h50m22.865247409s) [noise=true , address=45.91.116.8:4046 , piece_id=SJLZFPPOF6FT7RGPSORAHANJIWFDNVLNGWKGZJKOCO6DD5ZK5UDA, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.994419299 (627h50m22.865502335s) [noise=true , address=77.90.31.19:4844 , piece_id=YMHLUQ7B64YIGKICCWFHULLHQ2NW363YYDLMPGZTQY4U6F3X4RXA, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.994772873 (627h50m22.865855908s) [noise=true , address=45.91.116.4:4939 , piece_id=MX46IS7KFYCXIESW44ZU7RVINAFRUXG5ZLFIQ6T3EHEYHIQWFZSQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.995115766 (627h50m22.866198811s) [noise=true , address=145.79.227.21:4475 , piece_id=3J6ECTZXK35YKMK7N5NXVOXHD3NAAVFZQHLXZNJ2FI4V5IWOWBLQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.995424063 (627h50m22.866507098s) [noise=true , address=45.91.116.11:4117 , piece_id=5WUO63Z2JASDDGJW5JL4O7C6EC5IZ5AT4FHA7KUCXLZL3DDDCKPQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.995751105 (627h50m22.86683414s) [noise=true , address=145.79.227.6:4257 , piece_id=C5DFJN5QEOQYEFA4NWOWLIFJDZU5NEPFMF2YAV7VNF5LOPRIHC6Q, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.996160916 (627h50m22.867243951s) [noise=true , address=145.79.227.18:4007 , piece_id=VQYVYQC3EZF4MFBMX54KTK5PLZKWTJJJMOFRUE5R6OCVFU2KSQRQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.996389921 (627h50m22.867472956s) [noise=true , address=145.79.227.51:4410 , piece_id=FYVO7VF737WWOKXSTAZO4BJG4YCYLG2P3VUTPQTKX2DF6KXKGPIQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:22.997769519 (627h50m22.868852554s) [noise=true , address=145.79.227.42:4123 , piece_id=CS6J7U5CNJWN4CXHI3SW63DVEO5RQDIUNO7SS6JD3YJXIRF4KRRQ, seg_pos={0 0}, upload=550154]: Uploading piece... Dec 31 19:19:24.698003975 (627h50m24.56908701s) [upload=550154]: Next returned nil segment Dec 31 19:19:24.698052558 (627h50m24.569135603s) [upload=550154]: Waiting for error group managing segments... Dec 31 19:19:24.983664141 (627h50m24.854747176s) [noise=true , address=145.79.227.2:4114 , piece_id=KW65HIVKPRVWH7VBA2KCBDAHVTXVQZQAXV567ORQI56UYMBVOEBQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:24.984325740 (627h50m24.855408775s) [noise=true , address=145.79.227.42:4123 , piece_id=CS6J7U5CNJWN4CXHI3SW63DVEO5RQDIUNO7SS6JD3YJXIRF4KRRQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:24.992186699 (627h50m24.863269744s) [noise=true , address=145.79.227.21:4475 , piece_id=3J6ECTZXK35YKMK7N5NXVOXHD3NAAVFZQHLXZNJ2FI4V5IWOWBLQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.001915534 (627h50m24.872998569s) [noise=true , address=145.79.227.51:4410 , piece_id=FYVO7VF737WWOKXSTAZO4BJG4YCYLG2P3VUTPQTKX2DF6KXKGPIQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.021076939 (627h50m24.892159984s) [noise=true , address=145.79.227.50:4375 , piece_id=36HZINMYQ7NJNWA7RZXAGPIDPEE5FSBPMAOM6OVINFIYIDSU3XWQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.024330305 (627h50m24.89541334s) [noise=true , address=145.79.227.6:4257 , piece_id=C5DFJN5QEOQYEFA4NWOWLIFJDZU5NEPFMF2YAV7VNF5LOPRIHC6Q, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.067477523 (627h50m24.938560558s) [noise=true , address=145.79.227.37:4566 , piece_id=ELNTCOKPH2XDI4KWGHWISBZVEU44WEII2XGAZC3U6A4ICR65QU3A, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.068799060 (627h50m24.939882105s) [noise=true , address=145.79.227.23:4546 , piece_id=4LGWC7FMB2GPFU3ED2KGB3DRVJNRRDERBAPUAQNRMDAEYYMD2WZQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.094263497 (627h50m24.965346542s) [noise=true , address=145.79.227.18:4007 , piece_id=VQYVYQC3EZF4MFBMX54KTK5PLZKWTJJJMOFRUE5R6OCVFU2KSQRQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.156292799 (627h50m25.027375844s) [noise=true , address=77.90.31.19:4844 , piece_id=YMHLUQ7B64YIGKICCWFHULLHQ2NW363YYDLMPGZTQY4U6F3X4RXA, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.271768811 (627h50m25.142851846s) [noise=true , address=145.79.227.14:4838 , piece_id=RO74PIBDAVLNGO2FOX3WXJNCWRTT2VCXVMVDG5GJSBYN66YUGUSQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.313853575 (627h50m25.18493661s) [noise=true , address=45.91.116.8:4046 , piece_id=SJLZFPPOF6FT7RGPSORAHANJIWFDNVLNGWKGZJKOCO6DD5ZK5UDA, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.343589968 (627h50m25.214673003s) [noise=true , address=77.90.31.10:4520 , piece_id=E7KCAIUBEDLFUB3O7RESBSASNR6SBU3P62P6Y7LWKNGL6VY7PLLA, seg_pos={0 0}, upload=550154]: Done uploading piece. err: <nil> Dec 31 19:19:25.343609565 (627h50m25.21469261s) [seg_pos={0 0}, upload=550154]: Segment reached optimal threshold of 13 pieces. Dec 31 19:19:25.343828893 (627h50m25.214911938s) [noise=true , address=45.91.116.10:4081 , piece_id=GUHH5D6TIOPULXOVF6M2R3JWV7AHIYFIGHRF42PTHY7HEW4AVL6Q, seg_pos={0 0}, upload=550154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 19:19:25.343837169 (627h50m25.214920224s) [noise=true , address=45.91.116.10:4081 , piece_id=GUHH5D6TIOPULXOVF6M2R3JWV7AHIYFIGHRF42PTHY7HEW4AVL6Q, seg_pos={0 0}, upload=550154]: Long tail cancellation due to optimal threshold Dec 31 19:19:25.344040376 (627h50m25.215123411s) [noise=true , address=45.91.116.11:4117 , piece_id=5WUO63Z2JASDDGJW5JL4O7C6EC5IZ5AT4FHA7KUCXLZL3DDDCKPQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:56572->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:56572->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56572->45.91.116.11:4117: use of closed network connection Dec 31 19:19:25.344057498 (627h50m25.215140533s) [noise=true , address=45.91.116.11:4117 , piece_id=5WUO63Z2JASDDGJW5JL4O7C6EC5IZ5AT4FHA7KUCXLZL3DDDCKPQ, seg_pos={0 0}, upload=550154]: Long tail cancellation due to optimal threshold Dec 31 19:19:25.344107313 (627h50m25.215190348s) [noise=true , address=45.91.116.4:4939 , piece_id=MX46IS7KFYCXIESW44ZU7RVINAFRUXG5ZLFIQ6T3EHEYHIQWFZSQ, seg_pos={0 0}, upload=550154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:34316->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:34316->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34316->45.91.116.4:4939: use of closed network connection Dec 31 19:19:25.344114487 (627h50m25.215197532s) [noise=true , address=45.91.116.4:4939 , piece_id=MX46IS7KFYCXIESW44ZU7RVINAFRUXG5ZLFIQ6T3EHEYHIQWFZSQ, seg_pos={0 0}, upload=550154]: Long tail cancellation due to optimal threshold Dec 31 19:19:25.344170683 (627h50m25.215253718s) [seg_pos={0 0}, upload=550154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:19:25.344235397 (627h50m25.215318432s) [upload=550154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:19:25.344241989 (627h50m25.215325024s) [upload=550154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:19:25.639112933 (627h50m25.510195968s) [upload=550154]: Done uploading segments... Dec 31 19:19:25.639132281 (627h50m25.510215316s) [upload=550154]: Upload finished. err: <nil> Duration: 30.002157255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..................X.X................................X.] duration: 28.950532601s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 28.950559833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..................X.X................................X] duration: 1m0.004834832s trace: https://jaeger.dev.storj.io/trace/546a5e17f88592f1 FAIL (2025-12-31 19:19:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633433649 (627h49m6.504516684s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.633771102 (627h49m6.504854147s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634291613 (627h49m6.505374658s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.634804660 (627h49m6.505887695s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.635215542 (627h49m6.506298587s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.637174623 (627h49m6.508257658s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.638646856 (627h49m6.509729901s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.639918197 (627h49m6.511001232s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.641184909 (627h49m6.512267954s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.642615633 (627h49m6.513698679s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.644050335 (627h49m6.51513337s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.645512279 (627h49m6.516595324s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:06.646531239 (627h49m6.517614284s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Uploading piece... Dec 31 19:18:08.085968833 (627h49m7.957051868s) [upload=550135]: Next returned nil segment Dec 31 19:18:08.085985055 (627h49m7.95706809s) [upload=550135]: Waiting for error group managing segments... Dec 31 19:18:08.473719364 (627h49m8.344802399s) [noise=true , address=64.49.13.154:4929 , piece_id=MIEUW5G5ELVPDD6GO6HVO3WWPNWDNQT2MRB7J2TEFWNRWPUFVXLQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.589022677 (627h49m8.460105712s) [noise=true , address=145.79.227.16:4943 , piece_id=7NYY7JKUE7V6FHE42LXHM6YMI4YSHLAKQB5BTCPK76RXGWAMFUMQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.683112676 (627h49m8.554195721s) [noise=true , address=77.90.31.17:4782 , piece_id=MJ5IC2VZ2ZOXFLXKTUXLEIHSVYIMOXEWWGL6CVFTUK7TTZSSH35A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.764334223 (627h49m8.635417258s) [noise=true , address=77.90.31.21:4889 , piece_id=ZXT7APOAARBIWRQWHQATTXWS3IITHB4LXVTCQCRCRXVMMB36YHIA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.788109465 (627h49m8.6591925s) [noise=true , address=77.90.31.13:4643 , piece_id=DHSKJV3NUDMNA46NYWLJ6UTJKGXBPCC6LIJIJANIV3DDL7QTR56Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.842166296 (627h49m8.713249341s) [noise=true , address=45.91.116.12:4253 , piece_id=IQ7J2QBJ5ZQKA4NGYLWBOSUVLXGTBNSCPAZZRF5QM2RTIWZTBF2A, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:08.962657808 (627h49m8.833740843s) [noise=true , address=45.91.116.7:4036 , piece_id=ON6P6ZNZNRM7S2IOBZIGX27PPV2D2KA2IAHZZAJNXPHMXUTQLMHA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.376971527 (627h49m9.248054562s) [noise=true , address=145.79.227.33:4886 , piece_id=TK5JSMRVMOOJPYHQTU6W7BQVX557S6YSFYY3T7MRA755G4TTAJGA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.417560014 (627h49m9.288643059s) [noise=true , address=145.79.227.44:4222 , piece_id=7KBGF7QOEEEGVASL6LAGVVDOS37XAXN4N72JYV7BQPAKI2RUMPBQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.443103572 (627h49m9.314186607s) [noise=true , address=145.79.227.43:4168 , piece_id=SLKHHVVWX7TDSWMO7NYVX5HVPQEH3LVAMGHTW5KY3PCOHHFSYYVA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.485228043 (627h49m9.356311108s) [noise=true , address=145.79.227.49:4348 , piece_id=YLKODN4LBJNWDHVUH3COQ3BTYX6EZIODAKI5GONB4J3KHWZNFCNQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.541024376 (627h49m9.412107421s) [noise=true , address=145.79.227.40:4082 , piece_id=74DPI5S6BNBQMAHD7TOKXFU5ZPWOOQLUFDEZRYHVKW3WKLZYMZ2Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.748002251 (627h49m9.619085286s) [noise=true , address=145.79.227.35:4500 , piece_id=L52ZDVDVC4WWEXX7RSYIVIITEOAVNWGKRTZ5673TBQ6ZKFTPRUXA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914788745 (627h49m9.78587178s) [noise=true , address=145.79.227.29:4953 , piece_id=F3ARGFUER2IBF2FS6PV3ZMNOQYT6VJONJ2RU2GJ2E3EYA7CMDAPQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: <nil> Dec 31 19:18:09.914805728 (627h49m9.785888763s) [seg_pos={0 0}, upload=550135]: Segment reached optimal threshold of 14 pieces. Dec 31 19:18:09.914952978 (627h49m9.786036023s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX): protocol: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41702->77.90.31.11:4557: use of closed network connection Dec 31 19:18:09.914958268 (627h49m9.786041303s) [noise=true , address=77.90.31.11:4557 , piece_id=YRB5DEXTZED5POB54WYMCGGYT2W7UBKACCNPCUZ7AMWOJYCZG6KQ, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915124144 (627h49m9.786207179s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm): protocol: expected piece hash; context canceled; EOF Dec 31 19:18:09.915130797 (627h49m9.786213832s) [noise=true , address=145.79.227.4:4198 , piece_id=2FNJVYQDH7U2PCMAJYTFTDCLN6XFOQM7SKQPETK6QY4ZAZTIPU3Q, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915229064 (627h49m9.786312099s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60340->64.49.13.169:4705: use of closed network connection Dec 31 19:18:09.915258550 (627h49m9.786341585s) [noise=true , address=64.49.13.169:4705 , piece_id=YDS2HADFPDFGEXPHSDFE7VEZEQKHOAJGOGKV6ZSQVGROTZHD27SA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915396433 (627h49m9.786479478s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL): protocol: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46350->145.79.227.2:4141: use of closed network connection Dec 31 19:18:09.915418705 (627h49m9.78650178s) [noise=true , address=145.79.227.2:4141 , piece_id=7C5FDQFCXCIMBCNLJSCGH5N73NCYKHV3FNW2QPMAKOAKNZFUGDPA, seg_pos={0 0}, upload=550135]: Long tail cancellation due to optimal threshold Dec 31 19:18:09.915444304 (627h49m9.786527349s) [seg_pos={0 0}, upload=550135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:18:09.915501503 (627h49m9.786584538s) [upload=550135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:18:09.915506833 (627h49m9.786589868s) [upload=550135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:18:10.202346214 (627h49m10.073429249s) [upload=550135]: Done uploading segments... Dec 31 19:18:10.202370731 (627h49m10.073453776s) [upload=550135]: Upload finished. err: <nil> Duration: 1m0.004858617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X...X.X...XXX..X..X........X...XX..........XX.X..X.XX....] duration: 9.562299539s trace: https://jaeger.dev.storj.io/trace/426c52240509795c FAIL (2025-12-31 19:18:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ) context deadline exceeded ======LOGS====== Dec 31 19:16:03.138800523 (627h47m3.009883568s) [upload=550105]: Starting upload Dec 31 19:16:03.138867350 (627h47m3.009950385s) [upload=550105]: Done starting upload Dec 31 19:16:03.138940439 (627h47m3.010023484s) [upload=550105]: Uploading segments... Dec 31 19:16:03.139061561 (627h47m3.010144606s) [upload=550105]: Got next segment. Inline: false Dec 31 19:16:03.139094944 (627h47m3.010177989s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:16:03.139099392 (627h47m3.010182438s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:16:03.387310701 (627h47m3.258393746s) [seg_pos={0 0}, upload=550105]: Begin upload segment... Dec 31 19:16:03.387378881 (627h47m3.258461926s) [seg_pos={0 0}, upload=550105]: Done begin upload segment. Dec 31 19:16:03.387449616 (627h47m3.258532661s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387457531 (627h47m3.258540576s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387549396 (627h47m3.258632431s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388067172 (627h47m3.259150217s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388597902 (627h47m3.259680947s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389176835 (627h47m3.26025988s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389792196 (627h47m3.260875241s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.390552554 (627h47m3.261635589s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392194671 (627h47m3.263277716s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392358704 (627h47m3.263441749s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.393725817 (627h47m3.264808862s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.394004277 (627h47m3.265087322s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.395196587 (627h47m3.266279632s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396421520 (627h47m3.267504565s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396993379 (627h47m3.268076444s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.399508969 (627h47m3.270592004s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:05.115093765 (627h47m4.98617681s) [upload=550105]: Next returned nil segment Dec 31 19:16:05.115106840 (627h47m4.986189885s) [upload=550105]: Waiting for error group managing segments... Dec 31 19:16:05.398697366 (627h47m5.269780401s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.400181292 (627h47m5.271264327s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.407670954 (627h47m5.278753999s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.408133985 (627h47m5.27921703s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.415778902 (627h47m5.286861947s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.421332678 (627h47m5.292415723s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.426458237 (627h47m5.297541282s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.429416981 (627h47m5.300500026s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.495367770 (627h47m5.366450815s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.567594100 (627h47m5.438677145s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.703969578 (627h47m5.575052613s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.726221760 (627h47m5.597304795s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748099237 (627h47m5.619182272s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748118333 (627h47m5.619201368s) [seg_pos={0 0}, upload=550105]: Segment reached optimal threshold of 13 pieces. Dec 31 19:16:05.748308144 (627h47m5.619391179s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748316280 (627h47m5.619399315s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748443162 (627h47m5.619526197s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748449934 (627h47m5.619532979s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748586074 (627h47m5.619669109s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748595020 (627h47m5.619678065s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748616101 (627h47m5.619699146s) [seg_pos={0 0}, upload=550105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:16:05.748697095 (627h47m5.61978014s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:16:05.748704019 (627h47m5.619787064s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:16:06.041043515 (627h47m5.91212655s) [upload=550105]: Done uploading segments... Dec 31 19:16:06.041064244 (627h47m5.912147279s) [upload=550105]: Upload finished. err: <nil> Duration: 9.562330518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..................X.X................................] duration: 27.637455374s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:18:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 27.637474512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X...X.X...XXX..X..X........X...XX..........XX.X..X.XX...] duration: 6.023819041s trace: https://jaeger.dev.storj.io/trace/426c52240509795c FAIL (2025-12-31 19:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ) context deadline exceeded ======LOGS====== Dec 31 19:16:03.138800523 (627h47m3.009883568s) [upload=550105]: Starting upload Dec 31 19:16:03.138867350 (627h47m3.009950385s) [upload=550105]: Done starting upload Dec 31 19:16:03.138940439 (627h47m3.010023484s) [upload=550105]: Uploading segments... Dec 31 19:16:03.139061561 (627h47m3.010144606s) [upload=550105]: Got next segment. Inline: false Dec 31 19:16:03.139094944 (627h47m3.010177989s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:16:03.139099392 (627h47m3.010182438s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:16:03.387310701 (627h47m3.258393746s) [seg_pos={0 0}, upload=550105]: Begin upload segment... Dec 31 19:16:03.387378881 (627h47m3.258461926s) [seg_pos={0 0}, upload=550105]: Done begin upload segment. Dec 31 19:16:03.387449616 (627h47m3.258532661s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387457531 (627h47m3.258540576s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387549396 (627h47m3.258632431s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388067172 (627h47m3.259150217s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388597902 (627h47m3.259680947s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389176835 (627h47m3.26025988s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389792196 (627h47m3.260875241s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.390552554 (627h47m3.261635589s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392194671 (627h47m3.263277716s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392358704 (627h47m3.263441749s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.393725817 (627h47m3.264808862s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.394004277 (627h47m3.265087322s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.395196587 (627h47m3.266279632s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396421520 (627h47m3.267504565s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396993379 (627h47m3.268076444s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.399508969 (627h47m3.270592004s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:05.115093765 (627h47m4.98617681s) [upload=550105]: Next returned nil segment Dec 31 19:16:05.115106840 (627h47m4.986189885s) [upload=550105]: Waiting for error group managing segments... Dec 31 19:16:05.398697366 (627h47m5.269780401s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.400181292 (627h47m5.271264327s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.407670954 (627h47m5.278753999s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.408133985 (627h47m5.27921703s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.415778902 (627h47m5.286861947s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.421332678 (627h47m5.292415723s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.426458237 (627h47m5.297541282s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.429416981 (627h47m5.300500026s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.495367770 (627h47m5.366450815s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.567594100 (627h47m5.438677145s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.703969578 (627h47m5.575052613s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.726221760 (627h47m5.597304795s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748099237 (627h47m5.619182272s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748118333 (627h47m5.619201368s) [seg_pos={0 0}, upload=550105]: Segment reached optimal threshold of 13 pieces. Dec 31 19:16:05.748308144 (627h47m5.619391179s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748316280 (627h47m5.619399315s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748443162 (627h47m5.619526197s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748449934 (627h47m5.619532979s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748586074 (627h47m5.619669109s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748595020 (627h47m5.619678065s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748616101 (627h47m5.619699146s) [seg_pos={0 0}, upload=550105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:16:05.748697095 (627h47m5.61978014s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:16:05.748704019 (627h47m5.619787064s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:16:06.041043515 (627h47m5.91212655s) [upload=550105]: Done uploading segments... Dec 31 19:16:06.041064244 (627h47m5.912147279s) [upload=550105]: Upload finished. err: <nil> Duration: 6.023839772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..................X.X...............................] duration: 26.757500049s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 26.757515898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X...X.X...XXX..X..X........X...XX..........XX.X..X.XX..] duration: 5.909967096s trace: https://jaeger.dev.storj.io/trace/426c52240509795c FAIL (2025-12-31 19:17:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ) context deadline exceeded ======LOGS====== Dec 31 19:16:03.138800523 (627h47m3.009883568s) [upload=550105]: Starting upload Dec 31 19:16:03.138867350 (627h47m3.009950385s) [upload=550105]: Done starting upload Dec 31 19:16:03.138940439 (627h47m3.010023484s) [upload=550105]: Uploading segments... Dec 31 19:16:03.139061561 (627h47m3.010144606s) [upload=550105]: Got next segment. Inline: false Dec 31 19:16:03.139094944 (627h47m3.010177989s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:16:03.139099392 (627h47m3.010182438s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:16:03.387310701 (627h47m3.258393746s) [seg_pos={0 0}, upload=550105]: Begin upload segment... Dec 31 19:16:03.387378881 (627h47m3.258461926s) [seg_pos={0 0}, upload=550105]: Done begin upload segment. Dec 31 19:16:03.387449616 (627h47m3.258532661s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387457531 (627h47m3.258540576s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387549396 (627h47m3.258632431s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388067172 (627h47m3.259150217s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388597902 (627h47m3.259680947s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389176835 (627h47m3.26025988s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389792196 (627h47m3.260875241s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.390552554 (627h47m3.261635589s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392194671 (627h47m3.263277716s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392358704 (627h47m3.263441749s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.393725817 (627h47m3.264808862s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.394004277 (627h47m3.265087322s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.395196587 (627h47m3.266279632s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396421520 (627h47m3.267504565s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396993379 (627h47m3.268076444s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.399508969 (627h47m3.270592004s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:05.115093765 (627h47m4.98617681s) [upload=550105]: Next returned nil segment Dec 31 19:16:05.115106840 (627h47m4.986189885s) [upload=550105]: Waiting for error group managing segments... Dec 31 19:16:05.398697366 (627h47m5.269780401s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.400181292 (627h47m5.271264327s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.407670954 (627h47m5.278753999s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.408133985 (627h47m5.27921703s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.415778902 (627h47m5.286861947s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.421332678 (627h47m5.292415723s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.426458237 (627h47m5.297541282s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.429416981 (627h47m5.300500026s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.495367770 (627h47m5.366450815s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.567594100 (627h47m5.438677145s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.703969578 (627h47m5.575052613s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.726221760 (627h47m5.597304795s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748099237 (627h47m5.619182272s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748118333 (627h47m5.619201368s) [seg_pos={0 0}, upload=550105]: Segment reached optimal threshold of 13 pieces. Dec 31 19:16:05.748308144 (627h47m5.619391179s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748316280 (627h47m5.619399315s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748443162 (627h47m5.619526197s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748449934 (627h47m5.619532979s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748586074 (627h47m5.619669109s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748595020 (627h47m5.619678065s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748616101 (627h47m5.619699146s) [seg_pos={0 0}, upload=550105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:16:05.748697095 (627h47m5.61978014s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:16:05.748704019 (627h47m5.619787064s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:16:06.041043515 (627h47m5.91212655s) [upload=550105]: Done uploading segments... Dec 31 19:16:06.041064244 (627h47m5.912147279s) [upload=550105]: Upload finished. err: <nil> Duration: 5.90999073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..................X.X..............................] duration: 21.091370244s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:16:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 21.091403327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X...X.X...XXX..X..X........X...XX..........XX.X..X.XX.] duration: 6.027204957s trace: https://jaeger.dev.storj.io/trace/426c52240509795c FAIL (2025-12-31 19:16:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ) context deadline exceeded ======LOGS====== Dec 31 19:16:03.138800523 (627h47m3.009883568s) [upload=550105]: Starting upload Dec 31 19:16:03.138867350 (627h47m3.009950385s) [upload=550105]: Done starting upload Dec 31 19:16:03.138940439 (627h47m3.010023484s) [upload=550105]: Uploading segments... Dec 31 19:16:03.139061561 (627h47m3.010144606s) [upload=550105]: Got next segment. Inline: false Dec 31 19:16:03.139094944 (627h47m3.010177989s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:16:03.139099392 (627h47m3.010182438s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:16:03.387310701 (627h47m3.258393746s) [seg_pos={0 0}, upload=550105]: Begin upload segment... Dec 31 19:16:03.387378881 (627h47m3.258461926s) [seg_pos={0 0}, upload=550105]: Done begin upload segment. Dec 31 19:16:03.387449616 (627h47m3.258532661s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387457531 (627h47m3.258540576s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387549396 (627h47m3.258632431s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388067172 (627h47m3.259150217s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388597902 (627h47m3.259680947s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389176835 (627h47m3.26025988s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389792196 (627h47m3.260875241s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.390552554 (627h47m3.261635589s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392194671 (627h47m3.263277716s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392358704 (627h47m3.263441749s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.393725817 (627h47m3.264808862s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.394004277 (627h47m3.265087322s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.395196587 (627h47m3.266279632s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396421520 (627h47m3.267504565s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396993379 (627h47m3.268076444s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.399508969 (627h47m3.270592004s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:05.115093765 (627h47m4.98617681s) [upload=550105]: Next returned nil segment Dec 31 19:16:05.115106840 (627h47m4.986189885s) [upload=550105]: Waiting for error group managing segments... Dec 31 19:16:05.398697366 (627h47m5.269780401s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.400181292 (627h47m5.271264327s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.407670954 (627h47m5.278753999s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.408133985 (627h47m5.27921703s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.415778902 (627h47m5.286861947s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.421332678 (627h47m5.292415723s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.426458237 (627h47m5.297541282s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.429416981 (627h47m5.300500026s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.495367770 (627h47m5.366450815s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.567594100 (627h47m5.438677145s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.703969578 (627h47m5.575052613s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.726221760 (627h47m5.597304795s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748099237 (627h47m5.619182272s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748118333 (627h47m5.619201368s) [seg_pos={0 0}, upload=550105]: Segment reached optimal threshold of 13 pieces. Dec 31 19:16:05.748308144 (627h47m5.619391179s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748316280 (627h47m5.619399315s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748443162 (627h47m5.619526197s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748449934 (627h47m5.619532979s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748586074 (627h47m5.619669109s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748595020 (627h47m5.619678065s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748616101 (627h47m5.619699146s) [seg_pos={0 0}, upload=550105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:16:05.748697095 (627h47m5.61978014s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:16:05.748704019 (627h47m5.619787064s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:16:06.041043515 (627h47m5.91212655s) [upload=550105]: Done uploading segments... Dec 31 19:16:06.041064244 (627h47m5.912147279s) [upload=550105]: Upload finished. err: <nil> Duration: 6.02722706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....X...X.X...XXX..X..X........X...XX..........XX.X..X.XX] duration: 30.002849413s trace: https://jaeger.dev.storj.io/trace/426c52240509795c FAIL (2025-12-31 19:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ) context deadline exceeded ======LOGS====== Dec 31 19:16:03.138800523 (627h47m3.009883568s) [upload=550105]: Starting upload Dec 31 19:16:03.138867350 (627h47m3.009950385s) [upload=550105]: Done starting upload Dec 31 19:16:03.138940439 (627h47m3.010023484s) [upload=550105]: Uploading segments... Dec 31 19:16:03.139061561 (627h47m3.010144606s) [upload=550105]: Got next segment. Inline: false Dec 31 19:16:03.139094944 (627h47m3.010177989s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:16:03.139099392 (627h47m3.010182438s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:16:03.387310701 (627h47m3.258393746s) [seg_pos={0 0}, upload=550105]: Begin upload segment... Dec 31 19:16:03.387378881 (627h47m3.258461926s) [seg_pos={0 0}, upload=550105]: Done begin upload segment. Dec 31 19:16:03.387449616 (627h47m3.258532661s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387457531 (627h47m3.258540576s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.387549396 (627h47m3.258632431s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388067172 (627h47m3.259150217s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.388597902 (627h47m3.259680947s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389176835 (627h47m3.26025988s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.389792196 (627h47m3.260875241s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.390552554 (627h47m3.261635589s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392194671 (627h47m3.263277716s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.392358704 (627h47m3.263441749s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.393725817 (627h47m3.264808862s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.394004277 (627h47m3.265087322s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.395196587 (627h47m3.266279632s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396421520 (627h47m3.267504565s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.396993379 (627h47m3.268076444s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:03.399508969 (627h47m3.270592004s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Uploading piece... Dec 31 19:16:05.115093765 (627h47m4.98617681s) [upload=550105]: Next returned nil segment Dec 31 19:16:05.115106840 (627h47m4.986189885s) [upload=550105]: Waiting for error group managing segments... Dec 31 19:16:05.398697366 (627h47m5.269780401s) [noise=true , address=145.79.227.42:4121 , piece_id=RXD7PC4LY3DX7IMMSGIUS4CBGILI7LDQEBXDGKGRVXRXSD4UXCPQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.400181292 (627h47m5.271264327s) [noise=true , address=145.79.227.6:4259 , piece_id=XDBZZNW25AKLNMI33SBX2MNCYNIEHV76ULYX5F7MH6OZNLTVQ72Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.407670954 (627h47m5.278753999s) [noise=true , address=145.79.227.12:4767 , piece_id=DZWNEW7S2QWPMAAMIP7CF7YV4JOVA34Y5RXRXH447WMDYI256GBA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.408133985 (627h47m5.27921703s) [noise=true , address=145.79.227.3:4151 , piece_id=WJ7RWAXQPCRN4FPZSB67MAG7OVY5KJDEDEJOIGFBV2RF5YVY7ASQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.415778902 (627h47m5.286861947s) [noise=true , address=145.79.227.19:4079 , piece_id=KZ36VOUSELK5536MNW7LYOYZPYAHGBCB2BNKUBG3K65LXLGXFNNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.421332678 (627h47m5.292415723s) [noise=true , address=145.79.227.23:4546 , piece_id=X3AHYLWQFTPXCQJ5UMVBDYNQJZEXZKBYKY5O4YX7BKCSGSNXL2KQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.426458237 (627h47m5.297541282s) [noise=true , address=145.79.227.27:4690 , piece_id=CYBNWO65ZHS73TFTKGGAVNPSONDPK3TY23A3XHDV2W4SOA7AFARA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.429416981 (627h47m5.300500026s) [noise=true , address=145.79.227.22:4509 , piece_id=OXBMDGIX2JAEBGIO7SLRLKHUHP77TWVOMJAWA465IPQYHEF6JAVQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.495367770 (627h47m5.366450815s) [noise=true , address=145.79.227.24:4581 , piece_id=QEIAO5FG6WXHH3LU2QUMLXJDGEIBEE65M2CYLSKRNMZ6NHJL5EEQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.567594100 (627h47m5.438677145s) [noise=true , address=77.90.31.10:4517 , piece_id=MZ26AT5EGOZODOID5GOVH6OL6VRYHFX3ZP6AVAWLXHXQ3GVQ4PTQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.703969578 (627h47m5.575052613s) [noise=true , address=45.91.116.11:4119 , piece_id=NJ3HO22QPLFD32DRVKRODV3GJZYP2AY3WDKT36VJ2URX64OXJD6Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.726221760 (627h47m5.597304795s) [noise=true , address=45.91.116.5:4300 , piece_id=RKMKSHJLYEVFWQXN2VJUV7QWU3DXMCTHKYEVTYK3T6WR423D4RNA, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748099237 (627h47m5.619182272s) [noise=true , address=45.91.116.14:4192 , piece_id=EMU23PBN5MHPN33XJN52WPQXIFPI3DEGSGGSVHB3Z3CJ7QUA4ISQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: <nil> Dec 31 19:16:05.748118333 (627h47m5.619201368s) [seg_pos={0 0}, upload=550105]: Segment reached optimal threshold of 13 pieces. Dec 31 19:16:05.748308144 (627h47m5.619391179s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748316280 (627h47m5.619399315s) [noise=true , address=77.90.31.18:4805 , piece_id=RLRG7MJIQMV3RXWGAJDGR7N7ST2OLHEVP4VGR3Y4YZTFDSDBSTKQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748443162 (627h47m5.619526197s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748449934 (627h47m5.619532979s) [noise=true , address=45.91.116.9:4263 , piece_id=U6POY5WAHFWWGWZ4FQORKT5NUDIY2TUEC2NPXS7PQTHHROTVKDNQ, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748586074 (627h47m5.619669109s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:16:05.748595020 (627h47m5.619678065s) [noise=true , address=145.79.227.49:4339 , piece_id=ZY4KB3EOQBJ6NPBNY6DGI2C37PBMWMXXOWLAL2AJ76WID3S6AS4Q, seg_pos={0 0}, upload=550105]: Long tail cancellation due to optimal threshold Dec 31 19:16:05.748616101 (627h47m5.619699146s) [seg_pos={0 0}, upload=550105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:16:05.748697095 (627h47m5.61978014s) [upload=550105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:16:05.748704019 (627h47m5.619787064s) [upload=550105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:16:06.041043515 (627h47m5.91212655s) [upload=550105]: Done uploading segments... Dec 31 19:16:06.041064244 (627h47m5.912147279s) [upload=550105]: Upload finished. err: <nil> Duration: 30.002874722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..................X.X.............................] duration: 15.41095962s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:16:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 15.410988835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..................X.X............................] duration: 45.278438686s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:15:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 45.278468253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X....X...X.X...XXX..X..X........X...XX..........XX.X..X.X] duration: 30.001883168s trace: https://jaeger.dev.storj.io/trace/43ddcf5551613e1c FAIL (2025-12-31 19:15:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (627h46m6.63139758s) [upload=550092]: Uploading segments... Dec 31 19:15:06.760466034 (627h46m6.631549069s) [upload=550092]: Got next segment. Inline: false Dec 31 19:15:06.760526710 (627h46m6.631609745s) [upload=550092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:15:06.760532982 (627h46m6.631616017s) [upload=550092]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:15:07.007893301 (627h46m6.878976346s) [seg_pos={0 0}, upload=550092]: Begin upload segment... Dec 31 19:15:07.007952945 (627h46m6.87903599s) [seg_pos={0 0}, upload=550092]: Done begin upload segment. Dec 31 19:15:07.007987170 (627h46m6.879070215s) [noise=true , address=145.79.227.4:4185 , piece_id=IJSTPBFNT5CS5LO37FGUWPUW7KU5CRGDO7QIHONZRTBB7K3ISAPQ, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.008022046 (627h46m6.879105081s) [noise=true , address=145.79.227.10:4403 , piece_id=UJAAPQUGTPJY5ULKYENFLIJHS3ZMWL5CVJRVJV435YTIYBFYSOUQ, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.008589476 (627h46m6.879672521s) [noise=true , address=77.90.31.13:4627 , piece_id=SA3OELZQUOTM737GYSWR3HCY2T62EYUQ2DKAHR6GKBUXGUISAHGA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.009182275 (627h46m6.88026532s) [noise=true , address=45.91.116.7:4011 , piece_id=VBSFRQWTBGYXMMKDE4BVQGHMCVHTYTAPVHG5N6WFVZHU2ZMVTARA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.009581154 (627h46m6.880664189s) [noise=true , address=77.90.31.19:4844 , piece_id=WNOQUZ3R3S52LMWAYXQAOH4B52A7ZIAZEQXNHV4LCH43WK2UYR2A, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.010076437 (627h46m6.881159482s) [noise=true , address=145.79.227.2:4114 , piece_id=DIPMK37PTTVEMRDMCA5W3JJBVQF6PLXCLICXX4VSRK2RRBBVHU4A, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.010503621 (627h46m6.881586666s) [noise=true , address=145.79.227.25:4619 , piece_id=5UHHQUPDA4N6K2DIPOXD4SIIZWZ4YG6RXQX2IQQZ7DBFVDJ5LSFA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.010940643 (627h46m6.882023678s) [noise=true , address=145.79.227.41:4087 , piece_id=POQGVREPAYQ6SEB6Z6H4OVNFN3OOIHZCZ2YPOLJ2LLWO33VGEJSQ, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.011427059 (627h46m6.882510104s) [noise=true , address=145.79.227.33:4870 , piece_id=D2PWV3GHHGYMFJ6BHESDIKZPGJDRU6DFIMHLSNQC7IO7BUXMN3BA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.012486526 (627h46m6.883569571s) [noise=true , address=145.79.227.7:4295 , piece_id=5FKP56FWQBS7JVKBCGLQIXCCQN5TXGUJPJC2UBSDBIY2JKRU66MA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.014085731 (627h46m6.885168776s) [noise=true , address=145.79.227.47:4266 , piece_id=4PXO4BGAADHSWF7JBSOAZ6D2RXRZTWYT2M73EC3F3UTQT7CNYYGA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.016213994 (627h46m6.887297049s) [noise=true , address=145.79.227.19:4077 , piece_id=DXRXCJK6YZMSHUDNW53YR6PIAZUHBLUHLJLCPGTI5GDEWUB4EIBA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.018063165 (627h46m6.88914621s) [noise=true , address=45.91.116.12:4227 , piece_id=BDWQJKWKSFJBNJ6CBWBYBJIP4IQO3USOI2BC6MYBK6GZQKUTYXWA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.019760998 (627h46m6.890844043s) [noise=true , address=45.91.116.13:4153 , piece_id=MXYWIPARLMOSDMJW3JUYJTUQLYEZLYLB4ELLEBLUC5LFSKDXRUWA, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.021319987 (627h46m6.892403032s) [noise=true , address=145.79.227.51:4411 , piece_id=YQGLWDIIVG4QPH3MPEOVG2PE6ORZWCVRWVPE5BBIFQKCD6YLLHMQ, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:07.021616682 (627h46m6.892699727s) [noise=true , address=145.79.227.5:4221 , piece_id=PSVPNG6GTSHCQIGVDUCKUQQYTE24XYSZSHIY6ZU7WGO2WKDF6IFQ, seg_pos={0 0}, upload=550092]: Uploading piece... Dec 31 19:15:08.702082965 (627h46m8.57316601s) [upload=550092]: Next returned nil segment Dec 31 19:15:08.702101180 (627h46m8.573184225s) [upload=550092]: Waiting for error group managing segments... Dec 31 19:15:08.997135577 (627h46m8.868218612s) [noise=true , address=145.79.227.4:4185 , piece_id=IJSTPBFNT5CS5LO37FGUWPUW7KU5CRGDO7QIHONZRTBB7K3ISAPQ, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:08.998411256 (627h46m8.869494292s) [noise=true , address=145.79.227.19:4077 , piece_id=DXRXCJK6YZMSHUDNW53YR6PIAZUHBLUHLJLCPGTI5GDEWUB4EIBA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.011149411 (627h46m8.882232456s) [noise=true , address=145.79.227.51:4411 , piece_id=YQGLWDIIVG4QPH3MPEOVG2PE6ORZWCVRWVPE5BBIFQKCD6YLLHMQ, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.011994531 (627h46m8.883077576s) [noise=true , address=145.79.227.10:4403 , piece_id=UJAAPQUGTPJY5ULKYENFLIJHS3ZMWL5CVJRVJV435YTIYBFYSOUQ, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.012168052 (627h46m8.883251107s) [noise=true , address=145.79.227.2:4114 , piece_id=DIPMK37PTTVEMRDMCA5W3JJBVQF6PLXCLICXX4VSRK2RRBBVHU4A, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.014713108 (627h46m8.885796153s) [noise=true , address=145.79.227.25:4619 , piece_id=5UHHQUPDA4N6K2DIPOXD4SIIZWZ4YG6RXQX2IQQZ7DBFVDJ5LSFA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.022140170 (627h46m8.893223235s) [noise=true , address=145.79.227.47:4266 , piece_id=4PXO4BGAADHSWF7JBSOAZ6D2RXRZTWYT2M73EC3F3UTQT7CNYYGA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.027752617 (627h46m8.898835652s) [noise=true , address=145.79.227.5:4221 , piece_id=PSVPNG6GTSHCQIGVDUCKUQQYTE24XYSZSHIY6ZU7WGO2WKDF6IFQ, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.028328964 (627h46m8.89941201s) [noise=true , address=145.79.227.7:4295 , piece_id=5FKP56FWQBS7JVKBCGLQIXCCQN5TXGUJPJC2UBSDBIY2JKRU66MA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.063599314 (627h46m8.934682349s) [noise=true , address=145.79.227.33:4870 , piece_id=D2PWV3GHHGYMFJ6BHESDIKZPGJDRU6DFIMHLSNQC7IO7BUXMN3BA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.081996435 (627h46m8.95307948s) [noise=true , address=145.79.227.41:4087 , piece_id=POQGVREPAYQ6SEB6Z6H4OVNFN3OOIHZCZ2YPOLJ2LLWO33VGEJSQ, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.158476874 (627h46m9.029559959s) [noise=true , address=77.90.31.13:4627 , piece_id=SA3OELZQUOTM737GYSWR3HCY2T62EYUQ2DKAHR6GKBUXGUISAHGA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.188106884 (627h46m9.059189929s) [noise=true , address=77.90.31.19:4844 , piece_id=WNOQUZ3R3S52LMWAYXQAOH4B52A7ZIAZEQXNHV4LCH43WK2UYR2A, seg_pos={0 0}, upload=550092]: Done uploading piece. err: <nil> Dec 31 19:15:09.188141130 (627h46m9.059224175s) [seg_pos={0 0}, upload=550092]: Segment reached optimal threshold of 13 pieces. Dec 31 19:15:09.188372810 (627h46m9.059455855s) [noise=true , address=45.91.116.7:4011 , piece_id=VBSFRQWTBGYXMMKDE4BVQGHMCVHTYTAPVHG5N6WFVZHU2ZMVTARA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:48792->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:48792->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48792->45.91.116.7:4011: use of closed network connection Dec 31 19:15:09.188386577 (627h46m9.059469612s) [noise=true , address=45.91.116.7:4011 , piece_id=VBSFRQWTBGYXMMKDE4BVQGHMCVHTYTAPVHG5N6WFVZHU2ZMVTARA, seg_pos={0 0}, upload=550092]: Long tail cancellation due to optimal threshold Dec 31 19:15:09.188395954 (627h46m9.05947901s) [noise=true , address=45.91.116.13:4153 , piece_id=MXYWIPARLMOSDMJW3JUYJTUQLYEZLYLB4ELLEBLUC5LFSKDXRUWA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:60580->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:60580->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->45.91.116.13:4153: use of closed network connection Dec 31 19:15:09.188407386 (627h46m9.059490431s) [noise=true , address=45.91.116.13:4153 , piece_id=MXYWIPARLMOSDMJW3JUYJTUQLYEZLYLB4ELLEBLUC5LFSKDXRUWA, seg_pos={0 0}, upload=550092]: Long tail cancellation due to optimal threshold Dec 31 19:15:09.188638817 (627h46m9.059721862s) [noise=true , address=45.91.116.12:4227 , piece_id=BDWQJKWKSFJBNJ6CBWBYBJIP4IQO3USOI2BC6MYBK6GZQKUTYXWA, seg_pos={0 0}, upload=550092]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 19:15:09.188646662 (627h46m9.059729707s) [noise=true , address=45.91.116.12:4227 , piece_id=BDWQJKWKSFJBNJ6CBWBYBJIP4IQO3USOI2BC6MYBK6GZQKUTYXWA, seg_pos={0 0}, upload=550092]: Long tail cancellation due to optimal threshold Dec 31 19:15:09.188665588 (627h46m9.059748633s) [seg_pos={0 0}, upload=550092]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:15:09.188755118 (627h46m9.059838153s) [upload=550092]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:15:09.188761350 (627h46m9.059844395s) [upload=550092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:15:09.502272058 (627h46m9.373355103s) [upload=550092]: Done uploading segments... Dec 31 19:15:09.502296064 (627h46m9.373379099s) [upload=550092]: Upload finished. err: <nil> Duration: 30.001907655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..................X.X...........................] duration: 36.879819703s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 36.879836144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.X....X...X.X...XXX..X..X........X...XX..........XX.X..X.] duration: 6.079645455s trace: https://jaeger.dev.storj.io/trace/4b2794d51253fe6b FAIL (2025-12-31 19:14:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ) context deadline exceeded ======LOGS====== Dec 31 19:14:04.855848689 (627h45m4.726931734s) [upload=550073]: Starting upload Dec 31 19:14:04.855922449 (627h45m4.727005484s) [upload=550073]: Done starting upload Dec 31 19:14:04.856033340 (627h45m4.727116385s) [upload=550073]: Uploading segments... Dec 31 19:14:04.856218283 (627h45m4.727301338s) [upload=550073]: Got next segment. Inline: false Dec 31 19:14:04.856270502 (627h45m4.727353547s) [upload=550073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:14:04.856278497 (627h45m4.727361552s) [upload=550073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:14:05.101528538 (627h45m4.972611583s) [seg_pos={0 0}, upload=550073]: Begin upload segment... Dec 31 19:14:05.101599985 (627h45m4.97268303s) [seg_pos={0 0}, upload=550073]: Done begin upload segment. Dec 31 19:14:05.101677863 (627h45m4.972760898s) [noise=true , address=45.91.116.8:4045 , piece_id=URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.101701507 (627h45m4.972784553s) [noise=true , address=77.90.31.19:4844 , piece_id=F2TB74QPLGJVK5PPFYFHVPDP433B3KJRAWLDMTUAHVAF4N55J5KA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.101832407 (627h45m4.972915452s) [noise=true , address=145.79.227.50:4375 , piece_id=4TUTRL7RI4XXZBEXJQJGYZQ7FVG5LN6Y3HGOM2CAG3MVT2XL556Q, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.102283696 (627h45m4.973366741s) [noise=true , address=45.91.116.12:4228 , piece_id=YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.102716241 (627h45m4.973799286s) [noise=true , address=145.79.227.51:4411 , piece_id=EUPA57JODV42K5RM33NW6BTNVGUH743XDRM45GRQ7KPFQBA3HTCA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103216833 (627h45m4.974299878s) [noise=true , address=45.91.116.5:4298 , piece_id=NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103550559 (627h45m4.974633594s) [noise=true , address=145.79.227.20:4437 , piece_id=P3GPCK3TXRIXHUOQK7GUWLDC6QTCM7NJM4FPNCSOFPBBCTNMVO3A, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103855439 (627h45m4.974938484s) [noise=true , address=145.79.227.31:4798 , piece_id=Z2PEP6SSBJEBW2IZ2CM7TIDSCFHB6R57ECOZDKMPEJLAVPULP7GQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.104213351 (627h45m4.975296396s) [noise=true , address=145.79.227.8:4330 , piece_id=VLXU6UM7NOCQPN5AIUS7DVQ7AJMYM5F7NT3EY2ADM34IYYH22BYA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.104572134 (627h45m4.975655179s) [noise=true , address=145.79.227.52:4447 , piece_id=NWABAIGRQGNQDP3K72UQGTQUUS4DZTSR5PC7XDJG4BG3TJPEQXZA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105056186 (627h45m4.976139231s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105477318 (627h45m4.976560353s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105490703 (627h45m4.976573738s) [noise=true , address=145.79.227.15:4874 , piece_id=IQVJB4PHY7KZKNBLVBJ2G77LFYW3TFQPKTBZAXMBMB6VEWKVDHUQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.106074445 (627h45m4.97715749s) [noise=true , address=145.79.227.7:4295 , piece_id=WTSZHMP7VN25UYYIILIM57F37IKWT6AEAXY3YFNVR4CAFYSJWX2A, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.106618220 (627h45m4.977701266s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.107342649 (627h45m4.978425694s) [noise=true , address=145.79.227.45:4230 , piece_id=4WUPDSC4MDNLLT5NDKYYN32MZYRLFMMMQEAGEIPVCZKTGTV2U3RA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:06.801410105 (627h45m6.67249315s) [upload=550073]: Next returned nil segment Dec 31 19:14:06.801431124 (627h45m6.67251417s) [upload=550073]: Waiting for error group managing segments... Dec 31 19:14:07.093877784 (627h45m6.964960829s) [noise=true , address=145.79.227.20:4437 , piece_id=P3GPCK3TXRIXHUOQK7GUWLDC6QTCM7NJM4FPNCSOFPBBCTNMVO3A, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.096073646 (627h45m6.967156691s) [noise=true , address=145.79.227.7:4295 , piece_id=WTSZHMP7VN25UYYIILIM57F37IKWT6AEAXY3YFNVR4CAFYSJWX2A, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.098598975 (627h45m6.96968202s) [noise=true , address=145.79.227.8:4330 , piece_id=VLXU6UM7NOCQPN5AIUS7DVQ7AJMYM5F7NT3EY2ADM34IYYH22BYA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.101356676 (627h45m6.972439711s) [noise=true , address=145.79.227.31:4798 , piece_id=Z2PEP6SSBJEBW2IZ2CM7TIDSCFHB6R57ECOZDKMPEJLAVPULP7GQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.103815828 (627h45m6.974898873s) [noise=true , address=145.79.227.50:4375 , piece_id=4TUTRL7RI4XXZBEXJQJGYZQ7FVG5LN6Y3HGOM2CAG3MVT2XL556Q, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.105582622 (627h45m6.976665667s) [noise=true , address=145.79.227.45:4230 , piece_id=4WUPDSC4MDNLLT5NDKYYN32MZYRLFMMMQEAGEIPVCZKTGTV2U3RA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.116322192 (627h45m6.987405227s) [noise=true , address=145.79.227.15:4874 , piece_id=IQVJB4PHY7KZKNBLVBJ2G77LFYW3TFQPKTBZAXMBMB6VEWKVDHUQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.116988580 (627h45m6.988071625s) [noise=true , address=145.79.227.52:4447 , piece_id=NWABAIGRQGNQDP3K72UQGTQUUS4DZTSR5PC7XDJG4BG3TJPEQXZA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.132343348 (627h45m7.003426394s) [noise=true , address=145.79.227.51:4411 , piece_id=EUPA57JODV42K5RM33NW6BTNVGUH743XDRM45GRQ7KPFQBA3HTCA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.325665311 (627h45m7.196748346s) [noise=true , address=77.90.31.19:4844 , piece_id=F2TB74QPLGJVK5PPFYFHVPDP433B3KJRAWLDMTUAHVAF4N55J5KA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.392187047 (627h45m7.263270082s) [noise=true , address=45.91.116.5:4298 , piece_id=NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.406222293 (627h45m7.277305328s) [noise=true , address=45.91.116.8:4045 , piece_id=URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.459919289 (627h45m7.331002334s) [noise=true , address=45.91.116.12:4228 , piece_id=YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.459942734 (627h45m7.331025779s) [seg_pos={0 0}, upload=550073]: Segment reached optimal threshold of 13 pieces. Dec 31 19:14:07.460192620 (627h45m7.331275665s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460201969 (627h45m7.331285014s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460219782 (627h45m7.331302817s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460238368 (627h45m7.331321403s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460407018 (627h45m7.331490063s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460413892 (627h45m7.331496927s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460433619 (627h45m7.331516654s) [seg_pos={0 0}, upload=550073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:14:07.460525134 (627h45m7.331608169s) [upload=550073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:14:07.460530263 (627h45m7.331613298s) [upload=550073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:14:07.746905643 (627h45m7.617988698s) [upload=550073]: Done uploading segments... Dec 31 19:14:07.746945689 (627h45m7.618028734s) [upload=550073]: Upload finished. err: <nil> Duration: 6.079673319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX.X....X...X.X...XXX..X..X........X...XX..........XX.X..X] duration: 30.003128365s trace: https://jaeger.dev.storj.io/trace/4b2794d51253fe6b FAIL (2025-12-31 19:14:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ) context deadline exceeded ======LOGS====== Dec 31 19:14:04.855848689 (627h45m4.726931734s) [upload=550073]: Starting upload Dec 31 19:14:04.855922449 (627h45m4.727005484s) [upload=550073]: Done starting upload Dec 31 19:14:04.856033340 (627h45m4.727116385s) [upload=550073]: Uploading segments... Dec 31 19:14:04.856218283 (627h45m4.727301338s) [upload=550073]: Got next segment. Inline: false Dec 31 19:14:04.856270502 (627h45m4.727353547s) [upload=550073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:14:04.856278497 (627h45m4.727361552s) [upload=550073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:14:05.101528538 (627h45m4.972611583s) [seg_pos={0 0}, upload=550073]: Begin upload segment... Dec 31 19:14:05.101599985 (627h45m4.97268303s) [seg_pos={0 0}, upload=550073]: Done begin upload segment. Dec 31 19:14:05.101677863 (627h45m4.972760898s) [noise=true , address=45.91.116.8:4045 , piece_id=URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.101701507 (627h45m4.972784553s) [noise=true , address=77.90.31.19:4844 , piece_id=F2TB74QPLGJVK5PPFYFHVPDP433B3KJRAWLDMTUAHVAF4N55J5KA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.101832407 (627h45m4.972915452s) [noise=true , address=145.79.227.50:4375 , piece_id=4TUTRL7RI4XXZBEXJQJGYZQ7FVG5LN6Y3HGOM2CAG3MVT2XL556Q, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.102283696 (627h45m4.973366741s) [noise=true , address=45.91.116.12:4228 , piece_id=YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.102716241 (627h45m4.973799286s) [noise=true , address=145.79.227.51:4411 , piece_id=EUPA57JODV42K5RM33NW6BTNVGUH743XDRM45GRQ7KPFQBA3HTCA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103216833 (627h45m4.974299878s) [noise=true , address=45.91.116.5:4298 , piece_id=NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103550559 (627h45m4.974633594s) [noise=true , address=145.79.227.20:4437 , piece_id=P3GPCK3TXRIXHUOQK7GUWLDC6QTCM7NJM4FPNCSOFPBBCTNMVO3A, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.103855439 (627h45m4.974938484s) [noise=true , address=145.79.227.31:4798 , piece_id=Z2PEP6SSBJEBW2IZ2CM7TIDSCFHB6R57ECOZDKMPEJLAVPULP7GQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.104213351 (627h45m4.975296396s) [noise=true , address=145.79.227.8:4330 , piece_id=VLXU6UM7NOCQPN5AIUS7DVQ7AJMYM5F7NT3EY2ADM34IYYH22BYA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.104572134 (627h45m4.975655179s) [noise=true , address=145.79.227.52:4447 , piece_id=NWABAIGRQGNQDP3K72UQGTQUUS4DZTSR5PC7XDJG4BG3TJPEQXZA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105056186 (627h45m4.976139231s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105477318 (627h45m4.976560353s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.105490703 (627h45m4.976573738s) [noise=true , address=145.79.227.15:4874 , piece_id=IQVJB4PHY7KZKNBLVBJ2G77LFYW3TFQPKTBZAXMBMB6VEWKVDHUQ, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.106074445 (627h45m4.97715749s) [noise=true , address=145.79.227.7:4295 , piece_id=WTSZHMP7VN25UYYIILIM57F37IKWT6AEAXY3YFNVR4CAFYSJWX2A, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.106618220 (627h45m4.977701266s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:05.107342649 (627h45m4.978425694s) [noise=true , address=145.79.227.45:4230 , piece_id=4WUPDSC4MDNLLT5NDKYYN32MZYRLFMMMQEAGEIPVCZKTGTV2U3RA, seg_pos={0 0}, upload=550073]: Uploading piece... Dec 31 19:14:06.801410105 (627h45m6.67249315s) [upload=550073]: Next returned nil segment Dec 31 19:14:06.801431124 (627h45m6.67251417s) [upload=550073]: Waiting for error group managing segments... Dec 31 19:14:07.093877784 (627h45m6.964960829s) [noise=true , address=145.79.227.20:4437 , piece_id=P3GPCK3TXRIXHUOQK7GUWLDC6QTCM7NJM4FPNCSOFPBBCTNMVO3A, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.096073646 (627h45m6.967156691s) [noise=true , address=145.79.227.7:4295 , piece_id=WTSZHMP7VN25UYYIILIM57F37IKWT6AEAXY3YFNVR4CAFYSJWX2A, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.098598975 (627h45m6.96968202s) [noise=true , address=145.79.227.8:4330 , piece_id=VLXU6UM7NOCQPN5AIUS7DVQ7AJMYM5F7NT3EY2ADM34IYYH22BYA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.101356676 (627h45m6.972439711s) [noise=true , address=145.79.227.31:4798 , piece_id=Z2PEP6SSBJEBW2IZ2CM7TIDSCFHB6R57ECOZDKMPEJLAVPULP7GQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.103815828 (627h45m6.974898873s) [noise=true , address=145.79.227.50:4375 , piece_id=4TUTRL7RI4XXZBEXJQJGYZQ7FVG5LN6Y3HGOM2CAG3MVT2XL556Q, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.105582622 (627h45m6.976665667s) [noise=true , address=145.79.227.45:4230 , piece_id=4WUPDSC4MDNLLT5NDKYYN32MZYRLFMMMQEAGEIPVCZKTGTV2U3RA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.116322192 (627h45m6.987405227s) [noise=true , address=145.79.227.15:4874 , piece_id=IQVJB4PHY7KZKNBLVBJ2G77LFYW3TFQPKTBZAXMBMB6VEWKVDHUQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.116988580 (627h45m6.988071625s) [noise=true , address=145.79.227.52:4447 , piece_id=NWABAIGRQGNQDP3K72UQGTQUUS4DZTSR5PC7XDJG4BG3TJPEQXZA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.132343348 (627h45m7.003426394s) [noise=true , address=145.79.227.51:4411 , piece_id=EUPA57JODV42K5RM33NW6BTNVGUH743XDRM45GRQ7KPFQBA3HTCA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.325665311 (627h45m7.196748346s) [noise=true , address=77.90.31.19:4844 , piece_id=F2TB74QPLGJVK5PPFYFHVPDP433B3KJRAWLDMTUAHVAF4N55J5KA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.392187047 (627h45m7.263270082s) [noise=true , address=45.91.116.5:4298 , piece_id=NE6K5IG2JZT5UCTHT3D4JPCAQUE4UMIJRBOE5V7LA3ABP4KYNDXQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.406222293 (627h45m7.277305328s) [noise=true , address=45.91.116.8:4045 , piece_id=URXFLLIFJDGFIUYV54AEEPGQHZH6WBT5N5DRHTLRDGUTAVIS5APA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.459919289 (627h45m7.331002334s) [noise=true , address=45.91.116.12:4228 , piece_id=YQTIG7RK3EJWPAP25VHYZZ2K2FOKSTC27QETBSWF4SC56AV7TFHQ, seg_pos={0 0}, upload=550073]: Done uploading piece. err: <nil> Dec 31 19:14:07.459942734 (627h45m7.331025779s) [seg_pos={0 0}, upload=550073]: Segment reached optimal threshold of 13 pieces. Dec 31 19:14:07.460192620 (627h45m7.331275665s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460201969 (627h45m7.331285014s) [noise=true , address=145.79.227.40:4051 , piece_id=RAYQ6LWYEJEQPFG57EBC4SX7HGKLCXA2YL5EEE3IIC4XGEN2P64Q, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460219782 (627h45m7.331302817s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460238368 (627h45m7.331321403s) [noise=true , address=145.79.227.35:4481 , piece_id=LPZKNXGICDQLQLC2JDOFT4UH67Q44KVP6OYUH3A5Y6FEXRTRZLJA, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460407018 (627h45m7.331490063s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:14:07.460413892 (627h45m7.331496927s) [noise=true , address=45.91.116.4:4937 , piece_id=52YY3QD4T54SSHYSUK5M4K6QE2U7LB3C4U346ESI7ZKSPTRZQGQA, seg_pos={0 0}, upload=550073]: Long tail cancellation due to optimal threshold Dec 31 19:14:07.460433619 (627h45m7.331516654s) [seg_pos={0 0}, upload=550073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:14:07.460525134 (627h45m7.331608169s) [upload=550073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:14:07.460530263 (627h45m7.331613298s) [upload=550073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:14:07.746905643 (627h45m7.617988698s) [upload=550073]: Done uploading segments... Dec 31 19:14:07.746945689 (627h45m7.618028734s) [upload=550073]: Upload finished. err: <nil> Duration: 30.003152231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..................X.X..........................] duration: 19.711923247s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:14:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 19.711952802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX.X....X...X.X...XXX..X..X........X...XX..........XX.X..] duration: 6.287680605s trace: https://jaeger.dev.storj.io/trace/7b8f53863b0e5212 FAIL (2025-12-31 19:13:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA) context deadline exceeded ======LOGS====== 31 19:12:12.937870159 (627h43m12.808953224s) [upload=550046]: Starting upload Dec 31 19:12:12.937975690 (627h43m12.809058735s) [upload=550046]: Done starting upload Dec 31 19:12:12.938065520 (627h43m12.809148555s) [upload=550046]: Uploading segments... Dec 31 19:12:12.938169478 (627h43m12.809252513s) [upload=550046]: Got next segment. Inline: false Dec 31 19:12:12.938212300 (627h43m12.809295345s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:12:12.938217851 (627h43m12.809300886s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:12:13.188273532 (627h43m13.059356567s) [seg_pos={0 0}, upload=550046]: Begin upload segment... Dec 31 19:12:13.188330911 (627h43m13.059413947s) [seg_pos={0 0}, upload=550046]: Done begin upload segment. Dec 31 19:12:13.188365137 (627h43m13.059448172s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188380586 (627h43m13.059463651s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188829601 (627h43m13.059912646s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189292592 (627h43m13.060375627s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189592053 (627h43m13.060675088s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.190013024 (627h43m13.061096069s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.191371932 (627h43m13.062454987s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.192538694 (627h43m13.063621729s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.193086997 (627h43m13.064170032s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.195512586 (627h43m13.066595621s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.198372211 (627h43m13.069455256s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.200597870 (627h43m13.071680915s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.202038963 (627h43m13.073121998s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.203682483 (627h43m13.074765528s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209058690 (627h43m13.080141725s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209784613 (627h43m13.080867648s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:14.892738465 (627h43m14.7638215s) [upload=550046]: Next returned nil segment Dec 31 19:12:14.892752992 (627h43m14.763836037s) [upload=550046]: Waiting for error group managing segments... Dec 31 19:12:15.176606530 (627h43m15.047689565s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.181173336 (627h43m15.052256371s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.190874377 (627h43m15.061957413s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195436985 (627h43m15.06652002s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195906710 (627h43m15.066989745s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.212170969 (627h43m15.083254004s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.219212387 (627h43m15.090295432s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.280176551 (627h43m15.151259586s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.367647096 (627h43m15.238730141s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.377744614 (627h43m15.248827659s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.445989562 (627h43m15.317072607s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.489544436 (627h43m15.360627471s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520146427 (627h43m15.391229472s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520166866 (627h43m15.391249901s) [seg_pos={0 0}, upload=550046]: Segment reached optimal threshold of 13 pieces. Dec 31 19:12:15.520387485 (627h43m15.391470521s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520395792 (627h43m15.391478827s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520499369 (627h43m15.391582414s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520515129 (627h43m15.391598164s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520687247 (627h43m15.391770282s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection Dec 31 19:12:15.520692977 (627h43m15.391776012s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520716793 (627h43m15.391799828s) [seg_pos={0 0}, upload=550046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:12:15.520792868 (627h43m15.391875903s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:12:15.520797266 (627h43m15.391880301s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:12:15.819008533 (627h43m15.690091598s) [upload=550046]: Done uploading segments... Dec 31 19:12:15.819034413 (627h43m15.690117468s) [upload=550046]: Upload finished. err: <nil> Duration: 6.287704932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX.X....X...X.X...XXX..X..X........X...XX..........XX.X.] duration: 5.965363078s trace: https://jaeger.dev.storj.io/trace/7b8f53863b0e5212 FAIL (2025-12-31 19:13:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA) context deadline exceeded ======LOGS====== 31 19:12:12.937870159 (627h43m12.808953224s) [upload=550046]: Starting upload Dec 31 19:12:12.937975690 (627h43m12.809058735s) [upload=550046]: Done starting upload Dec 31 19:12:12.938065520 (627h43m12.809148555s) [upload=550046]: Uploading segments... Dec 31 19:12:12.938169478 (627h43m12.809252513s) [upload=550046]: Got next segment. Inline: false Dec 31 19:12:12.938212300 (627h43m12.809295345s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:12:12.938217851 (627h43m12.809300886s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:12:13.188273532 (627h43m13.059356567s) [seg_pos={0 0}, upload=550046]: Begin upload segment... Dec 31 19:12:13.188330911 (627h43m13.059413947s) [seg_pos={0 0}, upload=550046]: Done begin upload segment. Dec 31 19:12:13.188365137 (627h43m13.059448172s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188380586 (627h43m13.059463651s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188829601 (627h43m13.059912646s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189292592 (627h43m13.060375627s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189592053 (627h43m13.060675088s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.190013024 (627h43m13.061096069s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.191371932 (627h43m13.062454987s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.192538694 (627h43m13.063621729s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.193086997 (627h43m13.064170032s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.195512586 (627h43m13.066595621s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.198372211 (627h43m13.069455256s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.200597870 (627h43m13.071680915s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.202038963 (627h43m13.073121998s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.203682483 (627h43m13.074765528s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209058690 (627h43m13.080141725s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209784613 (627h43m13.080867648s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:14.892738465 (627h43m14.7638215s) [upload=550046]: Next returned nil segment Dec 31 19:12:14.892752992 (627h43m14.763836037s) [upload=550046]: Waiting for error group managing segments... Dec 31 19:12:15.176606530 (627h43m15.047689565s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.181173336 (627h43m15.052256371s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.190874377 (627h43m15.061957413s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195436985 (627h43m15.06652002s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195906710 (627h43m15.066989745s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.212170969 (627h43m15.083254004s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.219212387 (627h43m15.090295432s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.280176551 (627h43m15.151259586s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.367647096 (627h43m15.238730141s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.377744614 (627h43m15.248827659s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.445989562 (627h43m15.317072607s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.489544436 (627h43m15.360627471s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520146427 (627h43m15.391229472s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520166866 (627h43m15.391249901s) [seg_pos={0 0}, upload=550046]: Segment reached optimal threshold of 13 pieces. Dec 31 19:12:15.520387485 (627h43m15.391470521s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520395792 (627h43m15.391478827s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520499369 (627h43m15.391582414s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520515129 (627h43m15.391598164s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520687247 (627h43m15.391770282s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection Dec 31 19:12:15.520692977 (627h43m15.391776012s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520716793 (627h43m15.391799828s) [seg_pos={0 0}, upload=550046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:12:15.520792868 (627h43m15.391875903s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:12:15.520797266 (627h43m15.391880301s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:12:15.819008533 (627h43m15.690091598s) [upload=550046]: Done uploading segments... Dec 31 19:12:15.819034413 (627h43m15.690117468s) [upload=550046]: Upload finished. err: <nil> Duration: 5.965391742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..................X.X.........................] duration: 15.602843402s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 15.602858973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..................X.X........................] duration: 23.065293098s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:12:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 23.065307155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....XX.X....X...X.X...XXX..X..X........X...XX..........XX.X] duration: 30.003059167s trace: https://jaeger.dev.storj.io/trace/7b8f53863b0e5212 FAIL (2025-12-31 19:12:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA) context deadline exceeded ======LOGS====== 31 19:12:12.937870159 (627h43m12.808953224s) [upload=550046]: Starting upload Dec 31 19:12:12.937975690 (627h43m12.809058735s) [upload=550046]: Done starting upload Dec 31 19:12:12.938065520 (627h43m12.809148555s) [upload=550046]: Uploading segments... Dec 31 19:12:12.938169478 (627h43m12.809252513s) [upload=550046]: Got next segment. Inline: false Dec 31 19:12:12.938212300 (627h43m12.809295345s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:12:12.938217851 (627h43m12.809300886s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:12:13.188273532 (627h43m13.059356567s) [seg_pos={0 0}, upload=550046]: Begin upload segment... Dec 31 19:12:13.188330911 (627h43m13.059413947s) [seg_pos={0 0}, upload=550046]: Done begin upload segment. Dec 31 19:12:13.188365137 (627h43m13.059448172s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188380586 (627h43m13.059463651s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.188829601 (627h43m13.059912646s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189292592 (627h43m13.060375627s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.189592053 (627h43m13.060675088s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.190013024 (627h43m13.061096069s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.191371932 (627h43m13.062454987s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.192538694 (627h43m13.063621729s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.193086997 (627h43m13.064170032s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.195512586 (627h43m13.066595621s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.198372211 (627h43m13.069455256s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.200597870 (627h43m13.071680915s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.202038963 (627h43m13.073121998s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.203682483 (627h43m13.074765528s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209058690 (627h43m13.080141725s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:13.209784613 (627h43m13.080867648s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Uploading piece... Dec 31 19:12:14.892738465 (627h43m14.7638215s) [upload=550046]: Next returned nil segment Dec 31 19:12:14.892752992 (627h43m14.763836037s) [upload=550046]: Waiting for error group managing segments... Dec 31 19:12:15.176606530 (627h43m15.047689565s) [noise=true , address=145.79.227.6:4259 , piece_id=SJ3IFO6KM5V6BI3MTHFXTYQCGVUUTEWNIRJFPDLV6A4QLKDQLXKQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.181173336 (627h43m15.052256371s) [noise=true , address=145.79.227.39:4014 , piece_id=UCSMUCZXHDY5OSXA3Z7P2SQHGZS7DFP226OYWCA64XCAWHXX24DA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.190874377 (627h43m15.061957413s) [noise=true , address=145.79.227.8:4330 , piece_id=HUF5RFTGSAB3GDYDRWY6HLQRJ7ECUPPGW35ORYFJRUC6PPMXLCDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195436985 (627h43m15.06652002s) [noise=true , address=145.79.227.4:4187 , piece_id=EGBPSQUAQAMHH4TF73X42V2SWV6Q5XM2IGACU27F5A7I7PGVZD5Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.195906710 (627h43m15.066989745s) [noise=true , address=145.79.227.17:4972 , piece_id=SNYX2NHBDXS4ML3XQI75BIERTRGFLSXGPHOK5VIGPDWNBV3V3IZQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.212170969 (627h43m15.083254004s) [noise=true , address=145.79.227.23:4546 , piece_id=EPWV7YYPHSFBLOWNQNAMZEEBFON5FK6E4EQ2U4WUKQ2MLB2JPGHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.219212387 (627h43m15.090295432s) [noise=true , address=145.79.227.7:4295 , piece_id=BHJTGMF25PLOCRVNHP5EZG2D4XZ7E53GT4BAWFJCNZTCP6AMEDMA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.280176551 (627h43m15.151259586s) [noise=true , address=145.79.227.22:4511 , piece_id=7JYX4HZJFRTHHHOOW25ZEQS3R6DHQAGYRUFTYFIUONMCMSEL4LDQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.367647096 (627h43m15.238730141s) [noise=true , address=145.79.227.18:4007 , piece_id=6L7TP52B7BLDOH7IVJAEVS2DASE5B3KFQN4RSTISX7DVLTWEDQGQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.377744614 (627h43m15.248827659s) [noise=true , address=77.90.31.16:4735 , piece_id=S326NRD3HIMH3YUPMGU53HTZMYUHXK53O4RFR4DQL4NTLF2T3XUA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.445989562 (627h43m15.317072607s) [noise=true , address=77.90.31.13:4629 , piece_id=MEGXNII3FKSUKBR3FA6CQ6NMJ6XBXPDLPBLI3OQISS7HULE7WFSQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.489544436 (627h43m15.360627471s) [noise=true , address=45.91.116.5:4298 , piece_id=IKYMXJPIQ62KSXHPEPYTFYIN5DTAQDKYTYSKNSK7HXQATR6E2ZCQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520146427 (627h43m15.391229472s) [noise=true , address=45.91.116.7:4010 , piece_id=6S7GILTNUDXQL35BNYX2BZZPMDS4CKRHM3775D5JLVCR7QWHQNLA, seg_pos={0 0}, upload=550046]: Done uploading piece. err: <nil> Dec 31 19:12:15.520166866 (627h43m15.391249901s) [seg_pos={0 0}, upload=550046]: Segment reached optimal threshold of 13 pieces. Dec 31 19:12:15.520387485 (627h43m15.391470521s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520395792 (627h43m15.391478827s) [noise=true , address=45.91.116.14:4189 , piece_id=XWQ5BXGLTN3BE7J2L3VBWTB3DCCGROGVCYP5KHV6QSQCHYGG2H3Q, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520499369 (627h43m15.391582414s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 19:12:15.520515129 (627h43m15.391598164s) [noise=true , address=45.91.116.13:4154 , piece_id=2PRTMMH2ZGGGMSNNNROAVXTJ5F4CQ5NNJ66VVJWZOX7V3ZLLSYHQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520687247 (627h43m15.391770282s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38380->45.91.116.8:4046: use of closed network connection Dec 31 19:12:15.520692977 (627h43m15.391776012s) [noise=true , address=45.91.116.8:4046 , piece_id=46D64POLQ4JLP2LYCJR4FJ3BUAMTQTEOTBTSWZWFOLRZB6TBFVXQ, seg_pos={0 0}, upload=550046]: Long tail cancellation due to optimal threshold Dec 31 19:12:15.520716793 (627h43m15.391799828s) [seg_pos={0 0}, upload=550046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:12:15.520792868 (627h43m15.391875903s) [upload=550046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:12:15.520797266 (627h43m15.391880301s) [upload=550046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:12:15.819008533 (627h43m15.690091598s) [upload=550046]: Done uploading segments... Dec 31 19:12:15.819034413 (627h43m15.690117468s) [upload=550046]: Upload finished. err: <nil> Duration: 30.003080998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..................X.X.......................] duration: 34.740464772s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:12:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 34.740493097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..................X.X......................] duration: 22.519501854s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:11:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 22.51952114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....XX.X....X...X.X...XXX..X..X........X...XX..........XX.] duration: 5.950649288s trace: https://jaeger.dev.storj.io/trace/30c9af8294f6309b FAIL (2025-12-31 19:11:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27h41m28.733255396s) [upload=550019]: Done starting upload Dec 31 19:10:28.862403000 (627h41m28.733486035s) [upload=550019]: Uploading segments... Dec 31 19:10:28.862437927 (627h41m28.733520962s) [upload=550019]: Got next segment. Inline: false Dec 31 19:10:28.862481359 (627h41m28.733564394s) [upload=550019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:10:28.862486469 (627h41m28.733569504s) [upload=550019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:10:29.106399915 (627h41m28.97748295s) [seg_pos={0 0}, upload=550019]: Begin upload segment... Dec 31 19:10:29.106443097 (627h41m28.977526132s) [seg_pos={0 0}, upload=550019]: Done begin upload segment. Dec 31 19:10:29.106466923 (627h41m28.977549958s) [noise=true , address=145.79.227.38:4979 , piece_id=BMDSRLJ43VTGH47PBUHYGZPG2FGIG3KJ7SVTL3VCROGMQ3I7VVBA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.106533448 (627h41m28.977616494s) [noise=true , address=145.79.227.33:4871 , piece_id=Z3FEZDQPK6PGNKPJQBMT7AI5KH6AWODJVIGSSDJXNAA4UHCSHUTQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107223673 (627h41m28.978306708s) [noise=true , address=145.79.227.15:4874 , piece_id=AT4THXLAS3NY4RLFVD7UICXO3MR7LXYGNVAHYL5VQEQBNFB7BUUQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107518485 (627h41m28.97860153s) [noise=true , address=145.79.227.2:4114 , piece_id=J36JEEZNSLUEUZQX3GHYGYL7U6PNBTATKSRYFFV347O7MST5LR6A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107888869 (627h41m28.978971905s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108287980 (627h41m28.979371015s) [noise=true , address=77.90.31.12:4589 , piece_id=75RZEEQHTQVHVA76ITOMEOFM7R7BUUBLWFNZGNQJGCS5MHMRFLDQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108633829 (627h41m28.979716864s) [noise=true , address=145.79.227.13:4802 , piece_id=HMAGPSKUKMFJILWTAMCEQGFKRRYYQB6C5CDXT3OZRPSLB7JUAHBQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108914633 (627h41m28.979997678s) [noise=true , address=77.90.31.10:4519 , piece_id=J2W742HBGA6C7VAMXM3U3PCYMRFZOAYSRNPI6AH3A6B2726ZGJCQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.110061266 (627h41m28.981144301s) [noise=true , address=145.79.227.7:4295 , piece_id=RK2XDZEF2VR7PHYGOYSUOC3QAC7O5QWQKYBUKIMOH5K6FIZ3KPFQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.110632825 (627h41m28.98171586s) [noise=true , address=145.79.227.14:4839 , piece_id=I4NTAJJXVPZCUXIGAGW4QOMZJZYF6S6RCINNTBUHTNJEMZL6C72A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.111711970 (627h41m28.982795005s) [noise=true , address=45.91.116.13:4153 , piece_id=SPWLYRTF5PIKSFO7OQDC7SHFQFTRSK3EEOUNM34CZKSXQHOFED3Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.111751605 (627h41m28.98283464s) [noise=true , address=145.79.227.42:4121 , piece_id=QUTLJGVZY7NRCSIPD6X2LAQHAESSMH337AE4VJSCLCRGS7LAOV7Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.114221418 (627h41m28.985304453s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.115512316 (627h41m28.986595351s) [noise=true , address=145.79.227.3:4151 , piece_id=NSANTI4ZEZOXT32O4I35TV7OYQYJJHUF427WE6FLANRYDMH7PNAQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.116543640 (627h41m28.987626685s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.118303711 (627h41m28.989386746s) [noise=true , address=145.79.227.52:4447 , piece_id=X4LADRUGNR5AAIFRK3QPRMKYMJVSJZNZXIJQX74VWFDVWIECOG2Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:30.785563186 (627h41m30.656646221s) [upload=550019]: Next returned nil segment Dec 31 19:10:30.785575479 (627h41m30.656658514s) [upload=550019]: Waiting for error group managing segments... Dec 31 19:10:31.088632573 (627h41m30.959715608s) [noise=true , address=145.79.227.7:4295 , piece_id=RK2XDZEF2VR7PHYGOYSUOC3QAC7O5QWQKYBUKIMOH5K6FIZ3KPFQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.112826021 (627h41m30.983909056s) [noise=true , address=145.79.227.15:4874 , piece_id=AT4THXLAS3NY4RLFVD7UICXO3MR7LXYGNVAHYL5VQEQBNFB7BUUQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.118242144 (627h41m30.989325179s) [noise=true , address=145.79.227.52:4447 , piece_id=X4LADRUGNR5AAIFRK3QPRMKYMJVSJZNZXIJQX74VWFDVWIECOG2Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.119625277 (627h41m30.990708312s) [noise=true , address=145.79.227.3:4151 , piece_id=NSANTI4ZEZOXT32O4I35TV7OYQYJJHUF427WE6FLANRYDMH7PNAQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.123825446 (627h41m30.994908481s) [noise=true , address=145.79.227.2:4114 , piece_id=J36JEEZNSLUEUZQX3GHYGYL7U6PNBTATKSRYFFV347O7MST5LR6A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.177029434 (627h41m31.048112469s) [noise=true , address=145.79.227.42:4121 , piece_id=QUTLJGVZY7NRCSIPD6X2LAQHAESSMH337AE4VJSCLCRGS7LAOV7Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.185685866 (627h41m31.056768901s) [noise=true , address=145.79.227.38:4979 , piece_id=BMDSRLJ43VTGH47PBUHYGZPG2FGIG3KJ7SVTL3VCROGMQ3I7VVBA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.202427965 (627h41m31.073511s) [noise=true , address=145.79.227.14:4839 , piece_id=I4NTAJJXVPZCUXIGAGW4QOMZJZYF6S6RCINNTBUHTNJEMZL6C72A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.276400109 (627h41m31.147483144s) [noise=true , address=145.79.227.13:4802 , piece_id=HMAGPSKUKMFJILWTAMCEQGFKRRYYQB6C5CDXT3OZRPSLB7JUAHBQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.360282743 (627h41m31.231365778s) [noise=true , address=77.90.31.12:4589 , piece_id=75RZEEQHTQVHVA76ITOMEOFM7R7BUUBLWFNZGNQJGCS5MHMRFLDQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.383452813 (627h41m31.254535918s) [noise=true , address=145.79.227.33:4871 , piece_id=Z3FEZDQPK6PGNKPJQBMT7AI5KH6AWODJVIGSSDJXNAA4UHCSHUTQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.469007150 (627h41m31.340090185s) [noise=true , address=77.90.31.10:4519 , piece_id=J2W742HBGA6C7VAMXM3U3PCYMRFZOAYSRNPI6AH3A6B2726ZGJCQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.536363276 (627h41m31.407446311s) [noise=true , address=45.91.116.13:4153 , piece_id=SPWLYRTF5PIKSFO7OQDC7SHFQFTRSK3EEOUNM34CZKSXQHOFED3Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.536382983 (627h41m31.407466018s) [seg_pos={0 0}, upload=550019]: Segment reached optimal threshold of 13 pieces. Dec 31 19:10:31.536634391 (627h41m31.407717426s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 31 19:10:31.536643469 (627h41m31.407726514s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536838520 (627h41m31.407921575s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection Dec 31 19:10:31.536848961 (627h41m31.407932006s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536884579 (627h41m31.407967624s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:34834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:10:31.536891512 (627h41m31.407974547s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536912591 (627h41m31.407995636s) [seg_pos={0 0}, upload=550019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:10:31.537085110 (627h41m31.408168155s) [upload=550019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:10:31.537093827 (627h41m31.408176882s) [upload=550019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:10:31.822272006 (627h41m31.693355061s) [upload=550019]: Done uploading segments... Dec 31 19:10:31.822319357 (627h41m31.693402392s) [upload=550019]: Upload finished. err: <nil> Duration: 5.950667683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X....XX.X....X...X.X...XXX..X..X........X...XX..........XX] duration: 30.002644212s trace: https://jaeger.dev.storj.io/trace/30c9af8294f6309b FAIL (2025-12-31 19:10:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27h41m28.733255396s) [upload=550019]: Done starting upload Dec 31 19:10:28.862403000 (627h41m28.733486035s) [upload=550019]: Uploading segments... Dec 31 19:10:28.862437927 (627h41m28.733520962s) [upload=550019]: Got next segment. Inline: false Dec 31 19:10:28.862481359 (627h41m28.733564394s) [upload=550019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:10:28.862486469 (627h41m28.733569504s) [upload=550019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:10:29.106399915 (627h41m28.97748295s) [seg_pos={0 0}, upload=550019]: Begin upload segment... Dec 31 19:10:29.106443097 (627h41m28.977526132s) [seg_pos={0 0}, upload=550019]: Done begin upload segment. Dec 31 19:10:29.106466923 (627h41m28.977549958s) [noise=true , address=145.79.227.38:4979 , piece_id=BMDSRLJ43VTGH47PBUHYGZPG2FGIG3KJ7SVTL3VCROGMQ3I7VVBA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.106533448 (627h41m28.977616494s) [noise=true , address=145.79.227.33:4871 , piece_id=Z3FEZDQPK6PGNKPJQBMT7AI5KH6AWODJVIGSSDJXNAA4UHCSHUTQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107223673 (627h41m28.978306708s) [noise=true , address=145.79.227.15:4874 , piece_id=AT4THXLAS3NY4RLFVD7UICXO3MR7LXYGNVAHYL5VQEQBNFB7BUUQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107518485 (627h41m28.97860153s) [noise=true , address=145.79.227.2:4114 , piece_id=J36JEEZNSLUEUZQX3GHYGYL7U6PNBTATKSRYFFV347O7MST5LR6A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.107888869 (627h41m28.978971905s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108287980 (627h41m28.979371015s) [noise=true , address=77.90.31.12:4589 , piece_id=75RZEEQHTQVHVA76ITOMEOFM7R7BUUBLWFNZGNQJGCS5MHMRFLDQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108633829 (627h41m28.979716864s) [noise=true , address=145.79.227.13:4802 , piece_id=HMAGPSKUKMFJILWTAMCEQGFKRRYYQB6C5CDXT3OZRPSLB7JUAHBQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.108914633 (627h41m28.979997678s) [noise=true , address=77.90.31.10:4519 , piece_id=J2W742HBGA6C7VAMXM3U3PCYMRFZOAYSRNPI6AH3A6B2726ZGJCQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.110061266 (627h41m28.981144301s) [noise=true , address=145.79.227.7:4295 , piece_id=RK2XDZEF2VR7PHYGOYSUOC3QAC7O5QWQKYBUKIMOH5K6FIZ3KPFQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.110632825 (627h41m28.98171586s) [noise=true , address=145.79.227.14:4839 , piece_id=I4NTAJJXVPZCUXIGAGW4QOMZJZYF6S6RCINNTBUHTNJEMZL6C72A, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.111711970 (627h41m28.982795005s) [noise=true , address=45.91.116.13:4153 , piece_id=SPWLYRTF5PIKSFO7OQDC7SHFQFTRSK3EEOUNM34CZKSXQHOFED3Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.111751605 (627h41m28.98283464s) [noise=true , address=145.79.227.42:4121 , piece_id=QUTLJGVZY7NRCSIPD6X2LAQHAESSMH337AE4VJSCLCRGS7LAOV7Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.114221418 (627h41m28.985304453s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.115512316 (627h41m28.986595351s) [noise=true , address=145.79.227.3:4151 , piece_id=NSANTI4ZEZOXT32O4I35TV7OYQYJJHUF427WE6FLANRYDMH7PNAQ, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.116543640 (627h41m28.987626685s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:29.118303711 (627h41m28.989386746s) [noise=true , address=145.79.227.52:4447 , piece_id=X4LADRUGNR5AAIFRK3QPRMKYMJVSJZNZXIJQX74VWFDVWIECOG2Q, seg_pos={0 0}, upload=550019]: Uploading piece... Dec 31 19:10:30.785563186 (627h41m30.656646221s) [upload=550019]: Next returned nil segment Dec 31 19:10:30.785575479 (627h41m30.656658514s) [upload=550019]: Waiting for error group managing segments... Dec 31 19:10:31.088632573 (627h41m30.959715608s) [noise=true , address=145.79.227.7:4295 , piece_id=RK2XDZEF2VR7PHYGOYSUOC3QAC7O5QWQKYBUKIMOH5K6FIZ3KPFQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.112826021 (627h41m30.983909056s) [noise=true , address=145.79.227.15:4874 , piece_id=AT4THXLAS3NY4RLFVD7UICXO3MR7LXYGNVAHYL5VQEQBNFB7BUUQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.118242144 (627h41m30.989325179s) [noise=true , address=145.79.227.52:4447 , piece_id=X4LADRUGNR5AAIFRK3QPRMKYMJVSJZNZXIJQX74VWFDVWIECOG2Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.119625277 (627h41m30.990708312s) [noise=true , address=145.79.227.3:4151 , piece_id=NSANTI4ZEZOXT32O4I35TV7OYQYJJHUF427WE6FLANRYDMH7PNAQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.123825446 (627h41m30.994908481s) [noise=true , address=145.79.227.2:4114 , piece_id=J36JEEZNSLUEUZQX3GHYGYL7U6PNBTATKSRYFFV347O7MST5LR6A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.177029434 (627h41m31.048112469s) [noise=true , address=145.79.227.42:4121 , piece_id=QUTLJGVZY7NRCSIPD6X2LAQHAESSMH337AE4VJSCLCRGS7LAOV7Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.185685866 (627h41m31.056768901s) [noise=true , address=145.79.227.38:4979 , piece_id=BMDSRLJ43VTGH47PBUHYGZPG2FGIG3KJ7SVTL3VCROGMQ3I7VVBA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.202427965 (627h41m31.073511s) [noise=true , address=145.79.227.14:4839 , piece_id=I4NTAJJXVPZCUXIGAGW4QOMZJZYF6S6RCINNTBUHTNJEMZL6C72A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.276400109 (627h41m31.147483144s) [noise=true , address=145.79.227.13:4802 , piece_id=HMAGPSKUKMFJILWTAMCEQGFKRRYYQB6C5CDXT3OZRPSLB7JUAHBQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.360282743 (627h41m31.231365778s) [noise=true , address=77.90.31.12:4589 , piece_id=75RZEEQHTQVHVA76ITOMEOFM7R7BUUBLWFNZGNQJGCS5MHMRFLDQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.383452813 (627h41m31.254535918s) [noise=true , address=145.79.227.33:4871 , piece_id=Z3FEZDQPK6PGNKPJQBMT7AI5KH6AWODJVIGSSDJXNAA4UHCSHUTQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.469007150 (627h41m31.340090185s) [noise=true , address=77.90.31.10:4519 , piece_id=J2W742HBGA6C7VAMXM3U3PCYMRFZOAYSRNPI6AH3A6B2726ZGJCQ, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.536363276 (627h41m31.407446311s) [noise=true , address=45.91.116.13:4153 , piece_id=SPWLYRTF5PIKSFO7OQDC7SHFQFTRSK3EEOUNM34CZKSXQHOFED3Q, seg_pos={0 0}, upload=550019]: Done uploading piece. err: <nil> Dec 31 19:10:31.536382983 (627h41m31.407466018s) [seg_pos={0 0}, upload=550019]: Segment reached optimal threshold of 13 pieces. Dec 31 19:10:31.536634391 (627h41m31.407717426s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 31 19:10:31.536643469 (627h41m31.407726514s) [noise=true , address=77.90.31.17:4770 , piece_id=LCYYQIX4CERABNJ5MUJV23RIAS6MNR6PJVS7PGGKW3BFF3K6ZL7A, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536838520 (627h41m31.407921575s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42570->45.91.116.14:4192: use of closed network connection Dec 31 19:10:31.536848961 (627h41m31.407932006s) [noise=true , address=45.91.116.14:4192 , piece_id=YRODQ3RTXNNKK25RE2PEAV6UKOLVGIWVMLQPY2TCRLVGLDLXA2UA, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536884579 (627h41m31.407967624s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:34834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:10:31.536891512 (627h41m31.407974547s) [noise=true , address=45.91.116.12:4226 , piece_id=2X46GMVLG6FFYPAGZVQXX6DUQA46KFUWT3GMF34IRHZVJCN63PZA, seg_pos={0 0}, upload=550019]: Long tail cancellation due to optimal threshold Dec 31 19:10:31.536912591 (627h41m31.407995636s) [seg_pos={0 0}, upload=550019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:10:31.537085110 (627h41m31.408168155s) [upload=550019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:10:31.537093827 (627h41m31.408176882s) [upload=550019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:10:31.822272006 (627h41m31.693355061s) [upload=550019]: Done uploading segments... Dec 31 19:10:31.822319357 (627h41m31.693402392s) [upload=550019]: Upload finished. err: <nil> Duration: 30.00265384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..................X.X.....................] duration: 17.560583819s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:10:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 17.560610441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X....XX.X....X...X.X...XXX..X..X........X...XX..........X] duration: 30.002718649s trace: https://jaeger.dev.storj.io/trace/1e1a90f2d3d7a54f FAIL (2025-12-31 19:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: NPFBAUZZHCBZFZ6256STANLAD5O2ZGKV5RYMNPNODHPTBFDPGLBQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PNQ2SCVKBP37T2YQXIDUWEHNX5P5MLALFIGEJLAYF67YLPSTR2UQ) context deadline exceeded ======LOGS====== Got next segment. Inline: false Dec 31 19:09:58.662552385 (627h40m58.53363542s) [upload=550015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:09:58.662561833 (627h40m58.533644878s) [upload=550015]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:09:58.914303213 (627h40m58.785386258s) [seg_pos={0 0}, upload=550015]: Begin upload segment... Dec 31 19:09:58.914362817 (627h40m58.785622869s) [seg_pos={0 0}, upload=550015]: Done begin upload segment. Dec 31 19:09:58.914554593 (627h40m58.785637648s) [noise=true , address=145.79.227.52:4445 , piece_id=K6AQCLIJ75X2ADBTSFCCYOKSAFTR4CJS4ZTKIWLW7PFU5ZDDARVA, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.914602253 (627h40m58.785685298s) [noise=true , address=45.91.116.14:4191 , piece_id=NPFBAUZZHCBZFZ6256STANLAD5O2ZGKV5RYMNPNODHPTBFDPGLBQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.915011142 (627h40m58.786094187s) [noise=true , address=45.91.116.7:4009 , piece_id=PNQ2SCVKBP37T2YQXIDUWEHNX5P5MLALFIGEJLAYF67YLPSTR2UQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.915279212 (627h40m58.786362257s) [noise=true , address=145.79.227.38:4979 , piece_id=NGZD5ZENSIQPC273KSQCIM4F3D3MXGMRIA3J7C5LEV2RD2KCDRIQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.915915203 (627h40m58.786998239s) [noise=true , address=145.79.227.12:4765 , piece_id=WK5PPP3UDNTRHEAF6OANBCJQQCNUDE75SWDV4CQYQ7YEB5PZJWKQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.916332949 (627h40m58.787415984s) [noise=true , address=145.79.227.5:4222 , piece_id=O56LZEYTKMQS6BRL3TJZZFN2YI4SV3AQQEZ5ZUI7SCW4YC4PJE3Q, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.916813433 (627h40m58.787896468s) [noise=true , address=45.91.116.5:4300 , piece_id=JR77HMBOSOWWSPOHWON7X275GTXF4AW7AQ2SPEI4SRTDLF4MXOCQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.916860744 (627h40m58.787943779s) [noise=true , address=145.79.227.47:4265 , piece_id=JHCYRFHGN77KF2HJFAUIFRK2X2TIOXKFY4HMT3ZZBKTGWH623NRQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.917374552 (627h40m58.788457597s) [noise=true , address=145.79.227.7:4295 , piece_id=FKAWBGVZTDVPDVGR3FMLZ6TCTPYZ67KPXHC4BPRFOH5FZETVPI7Q, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.917911965 (627h40m58.788995s) [noise=true , address=145.79.227.48:4301 , piece_id=R2JXAVNMNMB4L7QHVUPLUT73OXWNMRTNTTYRMMCXGM73TYSIVOFA, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.918325503 (627h40m58.789408538s) [noise=true , address=45.91.116.12:4226 , piece_id=W5QNZQJWMLFSBTERHUZ5BHPLJTN6GPOLSNFF2W4EJDIBIS55X7HA, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.918795387 (627h40m58.789878422s) [noise=true , address=145.79.227.2:4113 , piece_id=BA3AFGXGGMBGSVG53TM7RMW4F5F6FZPEA5NOQ2HUDY26WXZZOPRQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.919203344 (627h40m58.790286389s) [noise=true , address=145.79.227.41:4087 , piece_id=FUDUWL4BH4P7B5JIZRILLZI66FM4MBIPJU2AZWZQJE3FOJK6WBCQ, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.919438873 (627h40m58.790521918s) [noise=true , address=45.91.116.4:4940 , piece_id=EXVPZBRS5URC3RUJ7BPNZR2TNQIT3UA7GZI3DDYEM3VIR23WVFRA, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.919561816 (627h40m58.790644852s) [noise=true , address=145.79.227.4:4186 , piece_id=CBPT2X4VKNYEGRJC7NOLLR3M76GR3V5V3IG3QKMXIHZUVNSQJZ5Q, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:09:58.919978209 (627h40m58.791061244s) [noise=true , address=77.90.31.18:4808 , piece_id=B4MCPHFGSVH7EKK6XBRKNHA7ISXC7ZTRJBIMW6WOM2MZ75EUE2FA, seg_pos={0 0}, upload=550015]: Uploading piece... Dec 31 19:10:00.609790932 (627h41m0.480874007s) [upload=550015]: Next returned nil segment Dec 31 19:10:00.609829326 (627h41m0.480912391s) [upload=550015]: Waiting for error group managing segments... Dec 31 19:10:00.890002596 (627h41m0.761085631s) [noise=true , address=145.79.227.52:4445 , piece_id=K6AQCLIJ75X2ADBTSFCCYOKSAFTR4CJS4ZTKIWLW7PFU5ZDDARVA, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.892057689 (627h41m0.763140724s) [noise=true , address=145.79.227.4:4186 , piece_id=CBPT2X4VKNYEGRJC7NOLLR3M76GR3V5V3IG3QKMXIHZUVNSQJZ5Q, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.898778977 (627h41m0.769862012s) [noise=true , address=145.79.227.48:4301 , piece_id=R2JXAVNMNMB4L7QHVUPLUT73OXWNMRTNTTYRMMCXGM73TYSIVOFA, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.899214827 (627h41m0.770297872s) [noise=true , address=145.79.227.5:4222 , piece_id=O56LZEYTKMQS6BRL3TJZZFN2YI4SV3AQQEZ5ZUI7SCW4YC4PJE3Q, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.914291836 (627h41m0.785374881s) [noise=true , address=145.79.227.12:4765 , piece_id=WK5PPP3UDNTRHEAF6OANBCJQQCNUDE75SWDV4CQYQ7YEB5PZJWKQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.915349439 (627h41m0.786432484s) [noise=true , address=145.79.227.38:4979 , piece_id=NGZD5ZENSIQPC273KSQCIM4F3D3MXGMRIA3J7C5LEV2RD2KCDRIQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.915824033 (627h41m0.786907068s) [noise=true , address=145.79.227.7:4295 , piece_id=FKAWBGVZTDVPDVGR3FMLZ6TCTPYZ67KPXHC4BPRFOH5FZETVPI7Q, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.921068138 (627h41m0.792151173s) [noise=true , address=145.79.227.47:4265 , piece_id=JHCYRFHGN77KF2HJFAUIFRK2X2TIOXKFY4HMT3ZZBKTGWH623NRQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:00.925762157 (627h41m0.796845192s) [noise=true , address=145.79.227.41:4087 , piece_id=FUDUWL4BH4P7B5JIZRILLZI66FM4MBIPJU2AZWZQJE3FOJK6WBCQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:01.057891359 (627h41m0.928974394s) [noise=true , address=145.79.227.2:4113 , piece_id=BA3AFGXGGMBGSVG53TM7RMW4F5F6FZPEA5NOQ2HUDY26WXZZOPRQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:01.247211370 (627h41m1.118294425s) [noise=true , address=45.91.116.14:4191 , piece_id=NPFBAUZZHCBZFZ6256STANLAD5O2ZGKV5RYMNPNODHPTBFDPGLBQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:01.290248459 (627h41m1.161331514s) [noise=true , address=45.91.116.7:4009 , piece_id=PNQ2SCVKBP37T2YQXIDUWEHNX5P5MLALFIGEJLAYF67YLPSTR2UQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:01.331480491 (627h41m1.202563536s) [noise=true , address=45.91.116.4:4940 , piece_id=EXVPZBRS5URC3RUJ7BPNZR2TNQIT3UA7GZI3DDYEM3VIR23WVFRA, seg_pos={0 0}, upload=550015]: Done uploading piece. err: <nil> Dec 31 19:10:01.331506029 (627h41m1.202589064s) [seg_pos={0 0}, upload=550015]: Segment reached optimal threshold of 13 pieces. Dec 31 19:10:01.331756156 (627h41m1.202839201s) [noise=true , address=45.91.116.12:4226 , piece_id=W5QNZQJWMLFSBTERHUZ5BHPLJTN6GPOLSNFF2W4EJDIBIS55X7HA, seg_pos={0 0}, upload=550015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:37084->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:37084->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37084->45.91.116.12:4226: use of closed network connection Dec 31 19:10:01.331766346 (627h41m1.202849391s) [noise=true , address=45.91.116.12:4226 , piece_id=W5QNZQJWMLFSBTERHUZ5BHPLJTN6GPOLSNFF2W4EJDIBIS55X7HA, seg_pos={0 0}, upload=550015]: Long tail cancellation due to optimal threshold Dec 31 19:10:01.331914277 (627h41m1.202997322s) [noise=true , address=45.91.116.5:4300 , piece_id=JR77HMBOSOWWSPOHWON7X275GTXF4AW7AQ2SPEI4SRTDLF4MXOCQ, seg_pos={0 0}, upload=550015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50754->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50754->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50754->45.91.116.5:4300: use of closed network connection Dec 31 19:10:01.331923875 (627h41m1.20300691s) [noise=true , address=45.91.116.5:4300 , piece_id=JR77HMBOSOWWSPOHWON7X275GTXF4AW7AQ2SPEI4SRTDLF4MXOCQ, seg_pos={0 0}, upload=550015]: Long tail cancellation due to optimal threshold Dec 31 19:10:01.332295293 (627h41m1.203378338s) [noise=true , address=77.90.31.18:4808 , piece_id=B4MCPHFGSVH7EKK6XBRKNHA7ISXC7ZTRJBIMW6WOM2MZ75EUE2FA, seg_pos={0 0}, upload=550015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:59752->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:10:01.332304069 (627h41m1.203387114s) [noise=true , address=77.90.31.18:4808 , piece_id=B4MCPHFGSVH7EKK6XBRKNHA7ISXC7ZTRJBIMW6WOM2MZ75EUE2FA, seg_pos={0 0}, upload=550015]: Long tail cancellation due to optimal threshold Dec 31 19:10:01.332326863 (627h41m1.203409908s) [seg_pos={0 0}, upload=550015]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:10:01.332409420 (627h41m1.203492465s) [upload=550015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:10:01.332417254 (627h41m1.203500289s) [upload=550015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:10:01.627277582 (627h41m1.498360637s) [upload=550015]: Done uploading segments... Dec 31 19:10:01.627305405 (627h41m1.49838846s) [upload=550015]: Upload finished. err: <nil> Duration: 30.002738436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..................X.X....................] duration: 30.325221228s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 30.325246256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..................X.X...................] duration: 25.719784295s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 25.719799453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X....XX.X....X...X.X...XXX..X..X........X...XX..........] duration: 5.774280248s trace: https://jaeger.dev.storj.io/trace/735677880864da8a FAIL (2025-12-31 19:08:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded ======LOGS====== segments... Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment... Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment. Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments... Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces. Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments... Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil> Duration: 5.77430214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..................X.X..................] duration: 24.692377592s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:08:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 24.692409113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..................X.X.................] duration: 33.249427386s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:08:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 33.249448215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX.........] duration: 5.926990989s trace: https://jaeger.dev.storj.io/trace/735677880864da8a FAIL (2025-12-31 19:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded ======LOGS====== segments... Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment... Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment. Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments... Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces. Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments... Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil> Duration: 5.927020736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX........] duration: 6.239143019s trace: https://jaeger.dev.storj.io/trace/735677880864da8a FAIL (2025-12-31 19:07:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded ======LOGS====== segments... Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment... Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment. Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments... Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces. Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments... Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil> Duration: 6.239170371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..................X.X................] duration: 29.614895148s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:07:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 29.614923522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.686656366s trace: https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4 FAIL (2025-12-31 19:07:16 UTC) orbiter warning: context deadline exceeded Duration: 3.686683818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX.......] duration: 6.20373339s trace: https://jaeger.dev.storj.io/trace/735677880864da8a FAIL (2025-12-31 19:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded ======LOGS====== segments... Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment... Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment. Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments... Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces. Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments... Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil> Duration: 6.203762696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.491856649s trace: https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4 FAIL (2025-12-31 19:06:22 UTC) orbiter warning: context deadline exceeded Duration: 3.491883831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..................X.X...............] duration: 31.149314338s trace: https://jaeger.dev.storj.io/trace/3fb4961d520cfd75 FAIL (2025-12-31 19:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded ======LOGS====== , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece... Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments... Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces. Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil> Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil> Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments... Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil> Duration: 31.149340116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX......] duration: 6.015054234s trace: https://jaeger.dev.storj.io/trace/735677880864da8a FAIL (2025-12-31 19:06:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded ======LOGS====== segments... Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment... Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment. Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece... Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments... Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil> Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces. Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments... Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil> Duration: 6.015072528s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)