Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.987493871s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:37:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.987521984s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X...X.........XX..X......................] duration: 5.85051264s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:36:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.850539761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.970926961s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:36:42 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.970958902s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.832646528s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:36:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.832708527s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X...X.........XX..X.....................] duration: 5.96512287s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.965164158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.974244383s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:35:57 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.974268339s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X...X.........XX..X....................] duration: 5.85530373s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:35:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.855333929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.957347163s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:35:21 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.957375957s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.586096528s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:35:20 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.586126446s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X...X.........XX..X...................] duration: 5.709992962s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:34:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.710021226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.092866265s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:34:34 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.092888687s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.711817329s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:34:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.711877111s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.006452517s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:33:49 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.006474638s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.824735928s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:33:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.824760596s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X...X.........XX..X..................] duration: 5.773662699s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.773692154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X...X.........XX..X.................] duration: 6.019527516s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:33:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 6.019556401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.709606837s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:33:00 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.709632958s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.999935039s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:32:59 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.999953984s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X...X.........XX..X................] duration: 5.828873145s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:32:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.828904784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.02100468s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:32:22 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.021047932s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X...X.........XX..X...............] duration: 6.028260627s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:32:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 6.028284774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.994585866s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:32:05 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.99464579s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.795210812s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:32:00 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.795237063s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X...X.........XX..X..............] duration: 5.856940467s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.856970614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.008755561s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:31:40 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.008786961s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X...X.........XX..X.............] duration: 5.719307537s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.719337283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X...X.........XX..X............] duration: 5.911216792s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.911246499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X...X.........XX..X...........] duration: 5.946545631s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.946593251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.714744358s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:30:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.714770447s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.955666213s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:30:42 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.955699375s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.001074949s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:30:26 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.001101319s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X...X.........XX..X..........] duration: 5.94243295s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:29:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.942463357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.746126526s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:29:35 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.746158415s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X...X.........XX..X.........] duration: 5.870371437s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:29:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.87039396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.978663698s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:29:21 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.978690038s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X...X.........XX..X........] duration: 5.839431399s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:29:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.839466083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.934922376s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:28:45 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.934952694s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.01653198s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:28:35 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.016564571s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X...X.........XX..X.......] duration: 5.901484827s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.901522719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X...X.........XX..X......] duration: 5.803324464s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.803356765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.974597947s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:27:48 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.974626772s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.829867791s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:27:45 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.829913278s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.8798142s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:27:28 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.879846923s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.982032043s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:27:27 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.982074994s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X...X.........XX..X.....] duration: 5.654842465s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.654874115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.967446251s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:27:01 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.967479113s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.654492759s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:26:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.65452452s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X...X.........XX..X....] duration: 5.813803726s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:26:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.813836267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.988164045s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:25:54 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.988193741s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X...X.........XX..X...] duration: 5.831497181s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.83152889s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)